Matrix Completion under Low-Rank Missing Mechanism

Covariance matrix plays an important role in risk management, asset pricing, and portfolio allocation. Covariance matrix estimation becomes challenging when the dimensionality is comparable or much larger than the sample size. A widely used approach for reducing dimensionality is based on multi-factor models. Although it has been well studied and quite successful in many applications, the quality of the estimated covariance matrix is often degraded due to a nontrivial amount of missing data in the factor matrix for both technical and cost reasons. Since the factor matrix is only approximately low rank or even has full rank, existing matrix completion algorithms are not applicable. We consider a new matrix completion paradigm using the factor models directly and apply the alternating direction method of multipliers for the recovery. Numerical experiments show that the nuclear-norm matrix completion approaches are not suitable but our proposed models and algorithms are promising.

In this paper, we compare and analyze clustering methods with missing data in health behavior research. In particular, we propose and analyze the use of compressive sensing's matrix completion along with spectral clustering to cluster health related data. The empirical tests and real data results show that these methods can outperform standard methods like LPA and FIML, in terms of lower misclassification rates in clustering and better matrix completion performance in missing data problems. According to our examination, a possible explanation of these improvements is that spectral clustering takes advantage of high data dimension and compressive sensing methods utilize the near-to-low-rank property of health data.

Objective To explore the optimal fitting path of missing data of the Scale to make the fitting data close to the real situation of patients' data. Methods Based on the complete data set of the SDS of 507 patients with stroke, the data simulation sets of Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR) were constructed by R software, respectively, with missing rates of 5%, 10%, 15%, 20%, 25%, 30%, 35%, and 40% under three missing mechanisms. Mean substitution (MS), random forest regression (RFR), and predictive mean matching (PMM) were used to fit the data. Root mean square error (RMSE), the width of 95% confidence intervals (95% CI), and Spearman correlation coefficient (SCC) were used to evaluate the fitting effect and determine the optimal fitting path. Results when dealing with the problem of missing data in scales, the optimal fitting path is ① under the MCAR deletion mechanism, when the deletion proportion is less than 20%, the MS method is the most convenient; when the missing ratio is greater than 20%, RFR algorithm is the best fitting method. ② Under the Mar mechanism, when the deletion ratio is less than 35%, the MS method is the most convenient. When the deletion ratio is greater than 35%, RFR has a better correlation. ③ Under the mechanism of MNAR, RFR is the best data fitting method, especially when the missing proportion is greater than 30%. In reality, when the deletion ratio is small, the complete case deletion method is the most commonly used, but the RFR algorithm can greatly expand the application scope of samples and save the cost of clinical research when the deletion ratio is less than 30%. The best way to deal with data missing should be based on the missing mechanism and proportion of actual data, and choose the best method between the statistical analysis ability of the research team, the effectiveness of the method, and the understanding of readers.

In this chapter, we discuss a problem of estimation of a large target matrix based 4 on a finite number of noisy measurements of linear functionals (often, random) 5 of this matrix. The underlying assumption is that the target matrix is of small 6 rank and the goal is to determine how the estimation error depends on the rank 7 as well as on other important parameters of the problem such as the number of 8 measurements and the variance of the noise. This problem can be viewed as a 9 natural noncommutative extension of sparse recovery problems discussed in the 10 previous chapters. As a matter of fact, low rank recovery is equivalent to sparse 11 recovery when all the matrices in question are diagonal. There are several important 12 instances of such problems, in particular, matrix completion [41, 45, 70, 123], 13 matrix regression [40, 90, 126] and the problem of density matrix estimation in 14 quantum state tomography [70, 71, 88]. We will study some of these problems 15 using general empirical processes techniques developed in the first several chapters. 16 Noncommutative Bernstein type inequalities established in Sect. 2.4 will play a 17 very special role in our analysis. The main results will be obtained for Hermitian 18 matrices. So called “Paulsen dilation” (see Sect. 2.4) can be then used to tackle 19 the case of rectangular matrices. Throughout the chapter, we use the notations 20 introduced in Sect. A.4.

In this thesis we study the problem of exact completion for 𝑚 × 𝑛 sized matrix of rank 𝑟 and the problem of low-rank estimation with the adaptive sampling method. We introduce a relation of the exact completion problem with the sparsest vector of column and row spaces. Using this relation, wepropose matrix completion algorithms that exactly recovers the target matrix. These algorithms are superior to previous works in two important ways. First, our algorithms exactly recovers 𝜇0-coherent column space matrices byprobability at least 1 − 𝜖 using much smaller observations complexity than - 𝒪(𝜇0𝑟𝑛log𝑟 𝜖 )—the state of art. Specifically, many of the previous adaptive sampling methods require to observe the entire matrix when the column space is highly coherent. However, we show that our method is still able to recover this type of matrices by observing a small fraction of entries under many scenarios. Second, we propose an exact completion algorithm, whichrequires minimal pre-information as either row or column space is not being highly coherent. We provide an extension of these algorithms that is robust to sparse random noise. Besides, we propose an additional low-rank estimationalgorithm that is robust to any small noise by adaptively studying the shape of column space. At the end of the thesis, we provide experimental results that illustrate the strength of the algorithms proposed here. This thesis have been written mainly based on the paper [12].

Matrix embedding is proved an effective way to improve embedding efficiency of steganography, and usually, higher dimensional matrix will provide better embedding efficiency. However, the sender might suffer huge computational complexity when employing high dimensional matrix. In this paper, a variation of matrix embedding is proposed. Instead of finding a coset leader as the modification of the cover image (which is the most time consuming step in matrix embedding), we turn to finding a vector in the coset which has relatively small Hamming weight. Such a vector can be found at reduced computational cost. Consequently, higher dimensional matrix can be used for practically implementable matrix embedding. The parity check matrix of random linear code is used in the experiments. It has shown that the proposed scheme can enhance the practicability and efficiency of the conventional matrix embedding.

The absence of some data values in any observed dataset has been a real hindrance to achieving valid results in statistical research. This paper aimed at the missing data widespread problem faced by analysts and statisticians in academia and professional environments. Some data-driven methods were studied to obtain accurate data. Projects that highly rely on data face this missing data problem. And since machine learning models are only as good as the data used to train them, the missing data problem has a real impact on the solutions developed for real-world problems. Therefore, in this dissertation, there is an attempt to solve this problem using different mechanisms. This is done by testing the effectiveness of both traditional and modern data imputation techniques by determining the loss of statistical power when these different approaches are used to tackle the missing data problem. At the end of this research dissertation, it should be easy to establish which methods are the best when handling the research problem. It is recommended that using Multivariate Imputation by Chained Equations (MICE) for MAR missingness is the best approach to dealing with missing data.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

This paper investigates limiting spectral properties of a high-dimensional sample spatial-sign covariance matrix when both the dimension of the observations and the sample size grow to infinity. The underlying population is general enough to include the popular independent components model and the family of elliptical distributions. The first result of the paper shows that the empirical spectral distribution of a high dimensional sample spatial-sign covariance matrix converges to a generalized Marčenko-Pastur distribution. Secondly, a new central limit theorem for a class of related linear spectral statistics is established.

This tutorial provides an exposition of a flexible geometric framework for high-dimensional estimation problems with constraints. The tutorial develops geometric intuition about high-dimensional sets, justifies it with some results of asymptotic convex geometry, and demonstrates connections between geometric results and estimation problems. The theory is illustrated with applications to sparse recovery, matrix completion, quantization, linear and logistic regression, and generalized linear models.KeywordsAsymptotic Convex GeometryHigh-dimensional Estimation ProblemSparse RecoveryExact Recovery ConditionGeneral Feasible SetsThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper presents a novel compressed sensing (CS) approach to high dimensional wireless channel estimation by optimizing the input to a deep generative network. Channel estimation using generative networks relies on the assumption that the reconstructed channel lies in the range of a generative model. Channel reconstruction using generative priors outperforms conventional CS techniques and requires fewer pilots. It also eliminates the need of a priori knowledge of the sparsifying basis, instead using the structure captured by the deep generative model as a prior. Using this prior, we also perform channel estimation from one-bit quantized pilot measurements, and propose a novel optimization objective function that attempts to maximize the correlation between the received signal and the generator's channel estimate while minimizing the rank of the channel estimate. Our approach significantly outperforms sparse signal recovery methods such as Orthogonal Matching Pursuit (OMP) and Approximate Message Passing (AMP) algorithms such as EM-GM-AMP for narrowband mmWave channel reconstruction, and its execution time is not noticeably affected by the increase in the number of received pilot symbols.

Due to the surge in COVID-19 cases, hospitals have had to receive many more patients than before, which has brought unprecedented pressure to the hospital system. Therefore, the emphasis of medical decision-making has shifted from reaching the best treatment effect to prioritizing the treatment of COVID-19 patients by hospitals, which is key to relieving the pressure on the hospital system and reducing the overall mortality rate of COVID-19. There is no doubt that establishing the prioritization of COVID-19 cases is fundamental and pivotal for hospitals to achieve the shift in medical decision-making. Prioritization of COVID-19 cases in previous studies was mostly based on one patient characteristic, mainly including age, health conditions, and gender. This paper focuses on two patient characteristics at the same time. The probability that a COVID-19 patient who died had a given health condition in a given age group is calculated using the matrix completion technique based on the high-rank assumption of Bayesian matrices and the properties of Markov matrices. The calculated results show that doctors should give patients over 55 with respiratory diseases, patients over 65 with circulatory diseases, and patients over 65 with diabetes a higher prioritization in COVID-19 treatment.

We consider the wavelet-based estimators of mean regression function with long memory moving average errors and investigate their asymptotic rates of convergence based on thresholding of empirical wavelet coefficients. We show that these estimators achieve nearly optimal minimax convergence rates within a logarithmic term over a large range of Besov function classes \(B^{s}_{p,q}\). Therefore, in the presence of long memory non-Gaussian moving average noise, wavelet estimators still achieve nearly optimal convergence rates and provide explicitly the extraordinary local adaptability. The theory is illustrated with some numerical examples.

Due to ethical and logistical concerns it is common for data monitoring committees to periodically monitor accruing clinical trial data to assess the safety, and possibly efficacy, of a new experimental treatment. When formalized, monitoring is typically implemented using group sequential methods. In some cases regulatory agencies have required that primary trial analyses should be based solely on the judgment of an independent review committee (IRC). The IRC assessments can produce difficulties for trial monitoring given the time lag typically associated with receiving assessments from the IRC. This results in a missing data problem wherein a surrogate measure of response may provide useful information for interim decisions and future monitoring strategies. In this paper, we present statistical tools that are helpful for monitoring a group sequential clinical trial with missing IRC data. We illustrate the proposed methodology in the case of binary endpoints under various missingness mechanisms including missing completely at random assessments and when missingness depends on the IRC’s measurement.

The high dimensional sparse matrix word representations applied with supervised non-parametric learning models can have some infractions from training the model and result in errors known as loss. This loss of information can be retained using mathematical functions known as Optimizers which calculate the partial derivatives of the model parameters to yield better results. In this paper we have presented a result oriented experimentation of loss convergence performed on 5000 dimensional sparse matrix word representation with L2 regularized non-parametric supervised text classification model. We have presented results over 10, 50 and 100 epochs of training and traced the optimization prowess of varied loss optimizers that use different convergence approaches.