An iterative approach to Rank Minimization Problems

An approximation theory of matrix rank minimization and its application to quadratic equations

Solving optimization problems with sparse or low-rank optimal solutions has been an important topic since the recent emergence of compressed sensing and its matrix extensions such as the matrix rank minimization and robust principal component analysis problems. Compressed sensing enables one to recover a signal or image with fewer observations than the “length” of the signal or image, and thus provides potential breakthroughs in applications where data acquisition is costly. However, the potential impact of compressed sensing cannot be realized without efficient optimization algorithms that can handle extremely large-scale and dense data from real applications. Although the convex relaxations of these problems can be reformulated as either linear programming, second-order cone programming or semidefinite programming problems, the standard methods for solving these relaxations are not applicable because the problems are usually of huge size and contain dense data. In this dissertation, we give efficient algorithms for solving these “sparse” optimization problems and analyze the convergence and iteration complexity properties of these algorithms. Chapter 2 presents algorithms for solving the linearly constrained matrix rank minimization problem. The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization. Although the latter can be cast and solved as a semidefinite programming problem, such an approach is computationally expensive when the matrices are large. In Chapter 2, we propose fixed-point and Bregman iterative algorithms for solving the nuclear norm minimization problem and prove convergence of the first of these algorithms. By using a homotopy approach together with an approximate singular value decomposition procedure, we get a very fast, robust and powerful algorithm, which we call FPCA (Fixed Point Continuation with Approximate SVD), that can solve very large matrix rank minimization problems. Our numerical results on randomly generated and real matrix completion problems demonstrate that this algorithm is much faster and provides much better recoverability than semidefinite programming solvers such as SDPT3. For example, our algorithm can recover 1000 × 1000 matrices of rank 50 with a relative error of 10−5 in about 3 minutes by sampling only 20 percent of the elements. We know of no other method that achieves as good recoverability. Numerical experiments on online recommendation, DNA microarray data set and image inpainting problems demonstrate the effectiveness of our algorithms. In Chapter 3, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving linearly constrained matrix rank minimization problems are reported. Chapters 4 and 5 considers alternating direction type methods for solving composite convex optimization problems. We present in Chapter 4 alternating linearization algorithms that are based on an alternating direction augmented Lagrangian approach for minimizing the sum of two convex functions. Our basic methods require at most O(1/e) iterations to obtain an e-optimal solution, while our accelerated (i.e., fast) versions require at most O(1/ 3 ) iterations, with little change in the computational effort required at each iteration. For more general problem, i.e., minimizing the sum of K convex functions, we propose multiple-splitting algorithms for solving them. We propose both basic and accelerated algorithms with O(1/e) and O(1/ 3 ) iteration complexity bounds for obtaining an e-optimal solution. To the best of our knowledge, the complexity results presented in these two chapters are the first ones of this type that have been given for splitting and alternating direction type methods. Numerical results on various applications in sparse and low-rank optimization, including compressed sensing, matrix completion, image deblurring, robust principal component analysis, are reported to demonstrate the efficiency of our methods.

The topic of the rank minimization problem with affine constraints has been well studied in recent years. However, in many applications the data can exhibit other structures beyond simply being low rank. For example, images and videos present complex spatio-temporal structures, which are largely ignored by current affine rank minimization (ARM) methods. In this paper, we propose a novel approximate message passing (AMP)-based approach that is capable of capturing additional structures in the matrix entries, and can be implemented in a wide range of applications with little or no modification. Using probabilistic low-rank factorization, we derive our generalized AMP-based algorithm as an approximation of the loopy belief propagation algorithm. In addition, we apply a rank selection strategy and an expectation-maximization estimation strategy that adaptively obtain the optimal value of the algorithmic parameters. Then, we discuss the specializations of our proposed algorithm to the applications of structured ARM problems, such as compressive hyperspectral imaging and compressive video surveillance. Simulation results with both synthetic and real data demonstrate that the proposed algorithm yields the state-of-the-art reconstruction performance while maintaining competitive computational complexity.

formula omitted] completions on partial matrices

This paper studies the problem of the restoration of images corrupted by mixed Gaussian-impulse noise. In recent years, low-rank matrix reconstruction has become a research hotspot in many scientific and engineering domains such as machine learning, image processing, computer vision and bioinformatics, which mainly involves the problem of matrix completion and robust principal component analysis, namely recovering a low-rank matrix from an incomplete but accurate sampling subset of its entries and from an observed data matrix with an unknown fraction of its entries being arbitrarily corrupted, respectively. Inspired by these ideas, we consider the problem of recovering a low-rank matrix from an incomplete sampling subset of its entries with an unknown fraction of the samplings contaminated by arbitrary errors, which is defined as the problem of matrix completion from corrupted samplings and modeled as a convex optimization problem that minimizes a combination of the nuclear norm and the -norm in this paper. Meanwhile, we put forward a novel and effective algorithm called augmented Lagrange multipliers to exactly solve the problem. For mixed Gaussian-impulse noise removal, we regard it as the problem of matrix completion from corrupted samplings, and restore the noisy image following an impulse-detecting procedure. Compared with some existing methods for mixed noise removal, the recovery quality performance of our method is dominant if images possess low-rank features such as geometrically regular textures and similar structured contents; especially when the density of impulse noise is relatively high and the variance of Gaussian noise is small, our method can outperform the traditional methods significantly not only in the simultaneous removal of Gaussian noise and impulse noise, and the restoration ability for a low-rank image matrix, but also in the preservation of textures and details in the image.

The spectral p-norm and nuclear p-norm of matrices and tensors appear in various applications, albeit both are NP-hard to compute. The former sets a foundation of [Formula: see text]-sphere-constrained polynomial optimization problems, and the latter has been found in many rank minimization problems in machine learning. We study approximation algorithms of the tensor nuclear p-norm with an aim to establish the approximation bound matching the best one of its dual norm, the tensor spectral p-norm. Driven by the application of sphere covering to approximate both tensor spectral and nuclear norms ([Formula: see text]), we propose several types of hitting sets that approximately represent the [Formula: see text]-sphere with adjustable parameters for different levels of approximations and cardinalities, providing an independent toolbox for decision making on [Formula: see text]-spheres. Using the idea in robust optimization and second-order cone programming, we obtain the first polynomial-time algorithm with an [Formula: see text]-approximation bound for the computation of the matrix nuclear p-norm when [Formula: see text] is a rational, paving a way for applications in modeling with the matrix nuclear p-norm. These two new results enable us to propose various polynomial-time approximation algorithms for the computation of the tensor nuclear p-norm using tensor partitions, convex optimization, and duality theory, attaining the same approximation bound to the best one of the tensor spectral p-norms. Effective performance of the proposed algorithms for the tensor nuclear p-norm is shown by numerical implementations. We believe the ideas of [Formula: see text]-sphere covering with its applications in approximating nuclear p-norm would be useful for tackling optimization problems on other sets, such as the binary hypercube with its applications in graph theory and neural networks and the nonnegative sphere with its applications in copositive programming and nonnegative matrix factorization. Funding: This work was supported by the National Natural Science Foundation of China [Grants 71825003, 72394360, 72394364, 72171141, 72192830, and 72192832], and the program for Innovative Research Team of Shanghai University of Finance and Economics.

This paper proposes a structured matrix rank minimization approach to a novel image inpainting. We utilize the autoregressive (AR) model to describe the gray level of image, and formulate the image inpainting problem as the signal recovery problem by estimating the model order. This problem is described as the rank minimization problem, which is NP hard in general. To solve the problem approximately, this paper proposes an algorithm utilizing the null space based alternating optimization (NSAO) algorithm. Numerical examples show that the proposed algorithm recovers missing pixels well.

This paper proposes an algorithm to compute the transmit beamformers for linear interference alignment for the MIMO interference channel and the MIMO interfering multiple-access/broadcast channel without symbol extensions. We first formulate the interference alignment problem as a rank minimization problem with linear constraints, then approximate the matrix rank by the nuclear norm. We further propose the use of an iterative reweighted nuclear norm approach and show that adaptive reweighting can significantly improve the algorithm's ability to find aligned beamformers. Simulation results show that the proposed algorithm is able to provide more interference-free dimensions and also converges faster than a previously proposed rank-constrained rank-minimization approach for interference alignment.

This paper addresses the problem of predicting future outputs of an unknown Linear Time Invariant System based solely on past input/output data corrupted by noise, and an a-priori bound on the system order. This situation arises in many scenarios of practical interest where an explicit a-priori model of the system is not available. The main result of the paper is a simple, computationally efficient tracking algorithm that does not entail identifying first the unknown dynamics. Rather, the problem of estimating the next value of the output is recast into a rank minimization problem and solved using some recently introduced convex relaxations. The potential of the proposed approach is illustrated using as an example the problem of tracking multiple targets in video sequences in the presence of occlusion.

It is always an important problem to recover sparse signals from observations corrupted by Gaussian noise and has been extensively investigated. In high-resolution maritime surveillance radars working at scan mode, ship classification, and recognition need to recover high-resolution range profiles (HRRPs) of ships from radar returns of several pulses with severe range sidelobe effect. Multipulse synthetic data by the aid of ship Doppler information are complex sparse signals corrupted by non-Gaussian correlated interference. In this article, a sparse recovery via iterative minimization (SRIM) method is proposed to estimate complex HRRPs of ships from multipulse synthetic data. The SRIM method adapts the non-Gaussianity nature of the interference in the multipulse synthetic data and ship complex HRRPs are modeled by the random sequences of the biparametric generalized Gaussian distributions (GGDs) (0 < <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> ≤ 1). In the SRIM method, the parameters of the GGD model are iteratively searched by the minimal criterion of the Kolmogorov–Smirnov distance of the residue and the interference model. The SRIM method is compared with the recent linear-programming-based method and the classic sparse learning via iterative minimization (SLIM) method by using simulated and measured radar data and the results show that the SRIM method obtains the better performance.

This paper presents a novel method for interest level estimation based on matrix completion via rank minimization. The proposed method estimates interest levels of target objects from human behavior features which are extracted during selecting these objects. Specifically, by adopting matrix completion via rank minimization, unknown interest levels can be estimated. Furthermore, the proposed method can also estimate unknown interest levels with some missing behavior features which are not correctly extracted by sensors. Experimental results show the effectiveness of the proposed method.

In this paper, the P0,1+-matrix completion is introduced. A real n × n matrix is a P0,1+-matrix if it is a P0+-matrix with all positive diagonal entries. Firstly some necessary conditions for P0,1+-matrix completion are obtained and later on some sufficient conditions are also shown. Also the relationship between the P0,1+-matrix completion problem and other completion problems of p-matrix classes are discussed. Finally most of the digraphs of order at most 4 having P0,1+-completion are classified.

Given an undirected graph G=(V,E) with node set V=[1,n], a subset $S\subseteq V$, and a rational vector $a\in {\rm\bf Q}^{S\cup E}$, the positive semidefinite matrix completion problem consists of determining whether there exists a real symmetric n × n positive semidefinite matrix X=(xij) satisfying xii=ai ($i\in S$) and xij=aij ($ij\in E$). Similarly, the Euclidean distance matrix completion problem asks for the existence of a Euclidean distance matrix completing a partially defined given matrix. It is not known whether these problems belong to NP. We show here that they can be solved in polynomial time when restricted to the graphs having a fixed minimum fill-in, the minimum fill-in of graph G being the minimum number of edges needed to be added to G in order to obtain a chordal graph. A simple combinatorial algorithm permits us to construct a completion in polynomial time in the chordal case. We also show that the completion problem is polynomially solvable for a class of graphs including wheels of fixed length (assuming all diagonal entries are specified). The running time of our algorithms is polynomially bounded in terms of n and the bitlength of the input a. We also observe that the matrix completion problem can be solved in polynomial time in the real number model for the class of graphs containing no homeomorph K4.

The matrix rank approximation has shown high effectiveness in the matrix rank minimization (MRM) problem, which aims to recover the underlying low-rank structure from the observed matrix by imposing the rank constraint. The nuclear norm, serving as a convex surrogate of matrix rank, is employed in the MRM problem by shrinking singular values of the observed entry. However, this substitution treats each singular value equally, which is virtually $ℓ_1$-norm penalty of the singular value vector. Theoretically, the rank function of the matrix can be considered as $ℓ_0$-norm of its singular values. Consequently, minimizing the nuclear norm frequently results in biased solutions in various applications. In this article, we first propose a novel nonconvex rank approximation, named tight and flexible rank (TFR) approximation, to describe rank function effectively. Specifically, the TFR approximation can more tightly approach the rank function and exhibit greater flexibility in handling diverse singular values, as compared to existing nonconvex rank approximations. Furthermore, we apply TFR approximation to matrix completion and develop a solving algorithm with guaranteed convergence based on the framework of proximal alternating minimization. Extensive experiments reveal that the proposed matrix completion model with TFR approximation outperforms several existing state-of-the-art convex and nonconvex methods.

We study the inverse problem of reproducing partially known second-order statistics of a linear time invariant system by the least number of possible input disturbance channels. This can be formulated as a rank minimization problem, and for its solution, we employ a convex relaxation based on the nuclear norm. The resulting optimization problem can be cast as a semi-definite program and solved efficiently using general-purpose solvers for small- and medium-size problems. In this paper, we focus on issues and techniques that are pertinent to large-scale systems. We bring in a re-parameterization which transforms the problem into a form suitable for the alternating direction method of multipliers. Furthermore, we show that each iteration of this algorithm amounts to solving a system of linear equations, an eigenvalue decomposition, and a singular value thresholding. An illustrative example is provided to demonstrate the effectiveness of the developed approach.