Algorithms for sparse and low-rank optimization: convergence, complexity and applications

The matrix rank minimization problem has applications in many fields, such as system identification, optimal control, low-dimensional embedding, etc. As this problem is NP-hard in general, its convex relaxation, the nuclear norm minimization problem, is often solved instead. Recently, Ma, Goldfarb and Chen proposed a fixed-point continuation algorithm for solving the nuclear norm minimization problem (Math. Program., doi: 10.1007/s10107-009-0306-5 , 2009). By incorporating an approximate singular value decomposition technique in this algorithm, the solution to the matrix rank minimization problem is usually obtained. In this paper, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving affinely constrained matrix rank minimization problems are reported.

The matrix rank approximation has shown high effectiveness in the matrix rank minimization (MRM) problem, which aims to recover the underlying low-rank structure from the observed matrix by imposing the rank constraint. The nuclear norm, serving as a convex surrogate of matrix rank, is employed in the MRM problem by shrinking singular values of the observed entry. However, this substitution treats each singular value equally, which is virtually $ℓ_1$-norm penalty of the singular value vector. Theoretically, the rank function of the matrix can be considered as $ℓ_0$-norm of its singular values. Consequently, minimizing the nuclear norm frequently results in biased solutions in various applications. In this article, we first propose a novel nonconvex rank approximation, named tight and flexible rank (TFR) approximation, to describe rank function effectively. Specifically, the TFR approximation can more tightly approach the rank function and exhibit greater flexibility in handling diverse singular values, as compared to existing nonconvex rank approximations. Furthermore, we apply TFR approximation to matrix completion and develop a solving algorithm with guaranteed convergence based on the framework of proximal alternating minimization. Extensive experiments reveal that the proposed matrix completion model with TFR approximation outperforms several existing state-of-the-art convex and nonconvex methods.

An approximation theory of matrix rank minimization and its application to quadratic equations

This paper provides an alternating optimization algorithm for large-scale matrix rank minimization problems and its parallel implementation on GPU. The matrix rank minimization problem has a lot of important applications in signal processing, and several useful algorithms have been proposed. However most algorithms cannot be applied to a large-scale problem because of high computational cost. This paper proposes a null space based algorithm, which provides a low-rank solution without computing inverse matrix nor singular value decomposition. The algorithm can be parallelized easily without any approximation and can be applied to a large-scale problem. Numerical examples show that the algorithm provides a low-rank solution efficiently and can be speed up by parallel GPU computing.

This letter presents a method of correcting misaligned seismograms scanned during the digitization process of analog seismograms. The proposed method uses the low-rank matrix recovery technique to seek a Euclidean transformation that can be used to implement the correction. As the rank of a matrix is a natural measure of regularity and symmetry of images, a misaligned seismogram is assumed to be corrected when the rank of the texture extracted from the seismogram itself reaches the minimum. Therefore, the misalignment correction problem can be considered as a matrix rank minimization problem. The augmented Lagrange multiplier is applied to solve this minimization problem because of its fast convergence. Compared with the traditional geometrical methods, our method works efficiently and conveniently as well as overcomes corruptions, such as notes, spots, or marks.

Compressive sensing (CS) has been applied to reduce the acquisition costs in the task of target localization by deploying a small number of sensors to collect measurements. It is based on a fact that the number of targets is much smaller than all possible positions where the targets could be located. To improve the localization accuracy with noisy measurements, our previous work has proposed a localization approach by solving a row-based least absolute shrinkage and selection operator (RLASSO) formulation, which utilizes the joint sparsity property. However, some practical concerns in localization implementation, e.g., a decrement in power radiated by targets and an enlargement of overall localization region, degrade the localization accuracy seriously. The performance impairment is caused by the common but undesirable limitations that the reduced radiation power and the enlarged field indeed make targets partially unobservable from individual sensor alone. To combat the impacts of such constrains on localization performance, this paper develops a cluster-based multi-target localization approach based on matrix rank minimization (MRM). The solutions representing all the collected measurements from distributed clusters are modeled to possess a low rank property. Capitalizing on this low rank property, a nuclear norm minimization problem is formulated to localize the positions of multiple targets. Simulation results reveal that the proposed localization approach based on MRM outperforms those via RLASSO and direct CS, since the low rank property enables efficient utilization of user diversity in localization even under partial observations.

In this paper, we propose a sparse coding algorithm based on matrix rank minimization and k-means clustering and for recognition. We consider the problem of removing the noise in the training samples and generating more samples at the same time. To accomplish this, we extended the matrix rank minimization problem to cope with complex data. Samples from the same class are segmented into several groups by k-means clustering algorithm, and matrix rank minimization is applied on the clustered data to separate the noises and recover the low-rank structures in the grouped data. An over-complete dictionary is constructed by connecting the low-rank structures and the training samples together to keep the samples diversity. Sparse representation is operated based on this over-complete dictionary for recognition. Furthermore, a parameter is introduced to adjust the weighting of the coefficients that code the noises. We apply the proposed algorithm for character and face recognition. Experiments with improved performances validate the effectiveness of the proposed algorithm.

The most important of skew correction for scanned document image is to estimate the skew angle. Traditional methods mostly based on its linear check, such as Hough transformation and so on. However, it is often affected by its texture structure or other noise. In this paper, a fast and robust skew estimation method is proposed based on low-rank matrix decomposition, which seeks an affine transformation that can be used to implement the correction. As the rank of a matrix is a natural measure of regularity and symmetry of images, a misaligned scanned document image is assumed to be correct when the rank of the texture extracted from the image itself is the minimum. Therefore, the skew correction problem can be considered as a matrix rank minimization problem. As experiment illustrated, our method works efficiently and robustly overcoming corruptions, such as lines, circles and so on.

In cognitive radio (CR) networks, multi-CR cooperation is required during spectrum sensing in order to cope with wireless fading effects and the hidden terminal problem. User cooperation offers not only channel diversity gain against fading, but also complexity gain in terms of reduced sampling costs per CR. The latter is particularly useful when the monitored spectrum has very wide bandwidth and yet individual CRs only have limited hardware capability. To jointly collect both diversity gain and complexity gain, this paper develops a novel cooperative spectrum sensing technique based on matrix rank minimization. Subject to sampling-rate limitations, CRs individually collect digital measurements from a segment of the wide spectrum via coordinated selective filtering, with optional compressive sampling to further reduce the sampling rates. The solutions representing the measurements of all users are modeled to possess a low-rank property, and the rank order is the same as the size of the nonzero support of the monitored wide spectrum. Accordingly, a nuclear norm minimization problem is formulated to jointly identify the nonzero support and hence the overall wideband spectrum occupancy. Both tradeoff evaluation and simulation results corroborate that the proposed cooperative sensing technique outperforms traditional averaging-based cooperative schemes given the same sampling costs, because the low-rank property enables efficient utilization and tradeoff of the user diversity in the absence of any channel knowledge.

Due to the technology scaling down, process variation has become a crucial challenge on both interconnect delay and reliability. To handle the process variation, statistical optimization has emerged as a popular technique for yield improvement. As a relatively new technique, second-order conic programming (SOCP) has recently attracted very much attention in the literature for statistical circuit optimization. However, we observe significant limitations of SOCP in its flexibility, accuracy, and scalability for statistical circuit optimization, especially when interconnects are considered. We thus present in this paper an effective and efficient alternative for multi-constrained statistical circuit optimization by both gate and wire sizing using Lagrangian relaxation (LR). Compared with SOCP, experimental results show that our LRbased algorithm can achieve much better solution quality by reducing 21% area and obtain 560X speed-up over SOCP.

Background/foreground separation is one of the most fundamental tasks in computer vision, especially for video data. Robust PCA (RPCA) and its tensor extension, namely, Robust Tensor PCA (RTPCA), provide an effective framework for background/foreground separation by decomposing the data into low-rank and sparse components, which contain the background and the foreground (moving objects), respectively. However, in real-world applications, the video data is contaminated with noise. For example, in metal additive manufacturing (AM), the processed X-ray video to study melt pool dynamics is very noisy. RPCA and RTPCA are not able to separate the background, foreground, and noise simultaneously. As a result, the noise will contaminate the background or the foreground or both. There is a need to remove the noise from the background and foreground. To achieve the three components decomposition, a smooth sparse Robust Tensor Decomposition (SS-RTD) model is proposed to decompose the data into static background, smooth foreground, and noise, respectively. Specifically, the static background is modeled by the low-rank tucker decomposition, the smooth foreground (moving objects) is modeled by the spatio-temporal continuity, which is enforced by the total variation regularization, and the noise is modeled by the sparsity, which is enforced by the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula> norm. An efficient algorithm based on alternating direction method of multipliers (ADMM) is implemented to solve the proposed model. Extensive experiments on both simulated and real data demonstrate that the proposed method significantly outperforms the state-of-the-art approaches for background/foreground separation in noisy cases. Note to Practitioners—This work is motivated by melt pool detection in metal additive manufacturing where the processed X-ray video from the monitoring system is very noisy. The objective is to recover the background with porosity defects and the foreground with melt pool in the presence of noise. Existing methods fail to separate the noise from the background and foreground since RPCA and RTPCA have only two components, which cannot explain the three components in the data. This paper puts forward a smooth sparse Robust Tensor Decomposition by decomposing the tensor data into low-rank, smooth, and sparse components, respectively. It is a highly effective method for background/foreground separation in noisy case. In the case studies on simulated video and X-ray data, the proposed method can handle non-additive noise, and even the case of high noise-ratio. In the proposed algorithm, there is only one tuning parameter <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda $ </tex-math></inline-formula> . Based on the case studies, our method achieves satisfying performance by taking any <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\lambda \in [{0.2,1}]$ </tex-math></inline-formula> with anisotropic total variation regularization. With this observation, practitioners can apply the proposed method without extensive parameter tuning work. Furthermore, the proposed method is also applicable to other popular industrial applications. Practitioners can use the proposed SS-RTD for degradation processes monitoring, where the degradation image contains the static background, anomaly, and random disturbance, respectively.

We investigate in this paper the generalized trust region subproblem (GTRS) of minimizing a general quadratic objective function subject to a general quadratic inequality constraint. By applying a simultaneous block diagonalization approach, we obtain a congruent canonical form for the symmetric matrices in both the objective and constraint functions. By exploiting the block separability of the canonical form, we show that all GTRSs with an optimal value bounded from below are second order cone programming (SOCP) representable. Our result generalizes the recent work of Ben-Tal and Hertog (Math. Program. 143(1-2):1-29, 2014), which establishes the SOCP representability of the GTRS under the assumption of the simultaneous diagonalizability of the two matrices in the objective and constraint functions. Compared with the state-of-the-art approach to reformulate the GTRS as a semi-definite programming problem, our SOCP reformulation delivers a much faster solution algorithm. We further extend our method to two variants of the GTRS in which the inequality constraint is replaced by either an equality constraint or an interval constraint. Our methods also enable us to obtain simplified versions of the classical S-lemma, the S-lemma with equality, and the S-lemma with interval bounds.

Principal Component Analysis (PCA) is one of the most important unsupervised methods to handle high-dimensional data. However, due to the high computational complexity of its eigen-decomposition solution, it is hard to apply PCA to the large-scale data with high dimensionality, e.g., millions of data points with millions of variables. Meanwhile, the squared L2-norm based objective makes it sensitive to data outliers. In recent research, the L1-norm maximization based PCA method was proposed for efficient computation and being robust to outliers. However, this work used a greedy strategy to solve the eigenvectors. Moreover, the L1-norm maximization based objective may not be the correct robust PCA formulation, because it loses the theoretical connection to the minimization of data reconstruction error, which is one of the most important intuitions and goals of PCA. In this paper, we propose to maximize the L21-norm based robust PCA objective, which is theoretically connected to the minimization of reconstruction error. More importantly, we propose the efficient non-greedy optimization algorithms to solve our objective and the more general L21-norm maximization problem with theoretically guaranteed convergence. Experimental results on real world data sets show the effectiveness of the proposed method for principal component analysis.

As a convex relaxation of the rank minimization model, the nuclear norm minimization (NNM) problem has been attracting significant research interest in recent years. The standard NNM regularizes each singular value equally, composing an easily calculated convex norm. However, this restricts its capability and flexibility in dealing with many practical problems, where the singular values have clear physical meanings and should be treated differently. In this paper we study the weighted nuclear norm minimization (WNNM) problem, which adaptively assigns weights on different singular values. As the key step of solving general WNNM models, the theoretical properties of the weighted nuclear norm proximal (WNNP) operator are investigated. Albeit nonconvex, we prove that WNNP is equivalent to a standard quadratic programming problem with linear constrains, which facilitates solving the original problem with off-the-shelf convex optimization solvers. In particular, when the weights are sorted in a non-descending order, its optimal solution can be easily obtained in closed-form. With WNNP, the solving strategies for multiple extensions of WNNM, including robust PCA and matrix completion, can be readily constructed under the alternating direction method of multipliers paradigm. Furthermore, inspired by the reweighted sparse coding scheme, we present an automatic weight setting method, which greatly facilitates the practical implementation of WNNM. The proposed WNNM methods achieve state-of-the-art performance in typical low level vision tasks, including image denoising, background subtraction and image inpainting.

RPCA (Robust Principal Component Analysis) recovers sparse and low rank components from the original observation data, RPCA commonly uses ADM (Alternate Direction Method) for solving, the efficiency of algorithm depends on the nuclear norm optimization solution, that is SVD. And the application of RPCA in computer vision, image and video and so on, large amounts of data, such as images and video, make it difficult for large-scale data SVD. In this paper, we used the random matrix algorithm to improve the SVD, respectively the Count Sketch algorithm, the Prototype Randomized k-SVD algorithm and the Faster Randomized k-SVD algorithm, The main idea is to reduce the size of the original large-scale data matrix and sample randomly. Using the random projection algorithm to obtain an approximation of the original matrix, and operate QR decomposition of the approximate matrix, get the unitary matrix corresponding to the approximate matrix, and do the corresponding SVD, finally we can get the results which is similar to the original matrix calculation. Obtaining approximation of the original data matrix. But the cost time and space have been greatly optimized. Simulation experiments based on single image and video foreground detection show that the proposed methods can greatly improve the efficiency of RPCA iterative optimization.