Exploiting low-rank structure in semidefinite programming by approximate operator splitting

Semidefinite programming (SDP) solvers are increasingly used as primitives in many program verification tasks to synthesize and verify polynomial invariants for a variety of systems including programs, hybrid systems and stochastic models. On one hand, they provide a tractable alternative to reasoning about semi-algebraic constraints. However, the results are often unreliable due to “numerical issues” that include a large number of reasons such as floating-point errors, ill-conditioned problems, failure of strict feasibility, and more generally, the specifics of the algorithms used to solve SDPs. These issues influence whether the final numerical results are trustworthy or not. In this paper, we briefly survey the emerging use of SDP solvers in the static analysis community. We report on the perils of using SDP solvers for common invariant synthesis tasks, characterizing the common failures that can lead to unreliable answers. Next, we demonstrate existing tools for guaranteed semidefinite programming that often prove inadequate to our needs. Finally, we present a solution for verified semidefinite programming that can be used to check the reliability of the solution output by the solver and a padding procedure that can check the presence of a feasible nearby solution to the one output by the solver. We report on some successful preliminary experiments involving our padding procedure.

In this paper, we consider a least square semidefinite programming problem under ellipsoidal data uncertainty. We show that the robustification of this uncertain problem can be reformulated as a semidefinite linear programming problem with an additional second-order cone constraint. We then provide an explicit quantitative sensitivity analysis on how the solution under the robustification depends on the size/shape of the ellipsoidal data uncertainty set. Next, we prove that, under suitable constraint qualifications, the reformulation has zero duality gap with its dual problem, even when the primal problem itself is infeasible. The dual problem is equivalent to minimizing a smooth objective function over the Cartesian product of second-order cones and the Euclidean space, which is easy to project onto. Thus, we propose a simple variant of the spectral projected gradient method (Birgin et al. in SIAM J. Optim. 10:1196---1211, 2000) to solve the dual problem. While it is well-known that any accumulation point of the sequence generated from the algorithm is a dual optimal solution, we show in addition that the dual objective value along the sequence generated converges to a finite value if and only if the primal problem is feasible, again under suitable constraint qualifications. This latter fact leads to a simple certificate for primal infeasibility in situations when the primal feasible set lies in a known compact set. As an application, we consider robust correlation stress testing where data uncertainty arises due to untimely recording of portfolio holdings. In our computational experiments on this particular application, our algorithm performs reasonably well on medium-sized problems for real data when finding the optimal solution (if exists) or identifying primal infeasibility, and usually outperforms the standard interior-point solver SDPT3 in terms of CPU time.

Semidefinite programming (SDP) covers a wide range of applications such as robust optimization, polynomial optimization, combinatorial optimization, system and control theory, financial engineering, machine learning, quantum information and quantum chemistry. In those applications, SDP problems can be large scale easily. Such large scale SDP problems often satisfy a certain sparsity characterized by a chordal graph structure. This sparsity is classified in two types. The one is the domain space sparsity (d-space sparsity) for positive semidefinite symmetric matrix variables involved in SDP problems, and the other the range space sparsity (r-space sparsity) for matrix-inequality constraints in SDP problems. In this short note, we survey how we exploit these two types of sparsities to solve large scale linear and nonlinear SDP problems. We refer to the paper [7] for more details.

The standard quadratic program (QPS) is minx??xTQx, where $$\Delta\subset\Re^n$$ is the simplex ? = {x ? 0 ? ?i=1n xi = 1}. QPS can be used to formulate combinatorial problems such as the maximum stable set problem, and also arises in global optimization algorithms for general quadratic programming when the search space is partitioned using simplices. One class of `d.c.' (for `difference between convex') bounds for QPS is based on writing Q=S?T, where S and T are both positive semidefinite, and bounding xT Sx (convex on ?) and ?xTx (concave on ?) separately. We show that the maximum possible such bound can be obtained by solving a semidefinite programming (SDP) problem. The dual of this SDP problem corresponds to adding a simple constraint to the well-known Shor relaxation of QPS. We show that the max d.c. bound is dominated by another known bound based on a copositive relaxation of QPS, also obtainable via SDP at comparable computational expense. We also discuss extensions of the d.c. bound to more general quadratic programming problems. For the application of QPS to bounding the stability number of a graph, we use a novel formulation of the Lovasz ? number to compare ?, Schrijver's ??, and the max d.c. bound.

The sensor network localization (SNL) problem in embedding dimension r consists of locating the positions of wireless sensors, given only the distances between sensors that are within radio range and the positions of a subset of the sensors (called anchors). Current solution techniques relax this problem to a weighted, nearest, (positive) semidefinite programming (SDP) completion problem by using the linear mapping between Euclidean distance matrices (EDM) and semidefinite matrices. The resulting SDP is solved using primal-dual interior point solvers, yielding an expensive and inexact solution. This relaxation is highly degenerate in the sense that the feasible set is restricted to a low dimensional face of the SDP cone, implying that the Slater constraint qualification fails. Cliques in the graph of the SNL problem give rise to this degeneracy in the SDP relaxation. In this paper, we take advantage of the absence of the Slater constraint qualification and derive a technique for the SNL problem, with exact data, that explicitly solves the corresponding rank restricted SDP problem. No SDP solvers are used. For randomly generated instances, we are able to efficiently solve many huge instances of this NP-hard problem to high accuracy by finding a representation of the minimal face of the SDP cone that contains the SDP matrix representation of the EDM. The main work of our algorithm consists in repeatedly finding the intersection of subspaces that represent the faces of the SDP cone that correspond to cliques of the SNL problem.

Semi-supervised kernel learning methods have been received much more attention in the past few years. Traditional semi-supervised Non-Parametric Kernel Learning (NPKL) methods usually formulate the learning task as a Semi-Definite Programming (SDP) problem, which is very time consuming. Although some fast semi-supervised NPKL methods have been proposed recently, they usually scale very poorly. Furthermore, many semi-supervised NPKL methods are developed based on the manifold assumption. But, such an assumption might be invalid when handling some high-dimensional and sparse data, which has severely negative effect on the performance of learning algorithms. In this paper, we propose a more efficient semi-supervised NPKL method, which can effectively learn a low-rank kernel matrix from must-link and cannot-link constraints. Specially, by virtue of the nonlinear spectral embedded technique based on extreme learning machine (ELM), the proposed method has the ability of coping with data points that do not have a clear manifold structure in a low dimensional space. The proposed method is formulated as a trace ratio optimization problem, which is combined with dimensionality reduction in ELM feature space and aims to find optimal low-rank kernel matrices. The proposed optimization problem can be solved much more efficiently than SDP solvers. Extensive experiments have validated the superior performance of the proposed method compared to state-of-the-art semi-supervised kernel learning methods.

Solving optimization problems with sparse or low-rank optimal solutions has been an important topic since the recent emergence of compressed sensing and its matrix extensions such as the matrix rank minimization and robust principal component analysis problems. Compressed sensing enables one to recover a signal or image with fewer observations than the “length” of the signal or image, and thus provides potential breakthroughs in applications where data acquisition is costly. However, the potential impact of compressed sensing cannot be realized without efficient optimization algorithms that can handle extremely large-scale and dense data from real applications. Although the convex relaxations of these problems can be reformulated as either linear programming, second-order cone programming or semidefinite programming problems, the standard methods for solving these relaxations are not applicable because the problems are usually of huge size and contain dense data. In this dissertation, we give efficient algorithms for solving these “sparse” optimization problems and analyze the convergence and iteration complexity properties of these algorithms. Chapter 2 presents algorithms for solving the linearly constrained matrix rank minimization problem. The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization. Although the latter can be cast and solved as a semidefinite programming problem, such an approach is computationally expensive when the matrices are large. In Chapter 2, we propose fixed-point and Bregman iterative algorithms for solving the nuclear norm minimization problem and prove convergence of the first of these algorithms. By using a homotopy approach together with an approximate singular value decomposition procedure, we get a very fast, robust and powerful algorithm, which we call FPCA (Fixed Point Continuation with Approximate SVD), that can solve very large matrix rank minimization problems. Our numerical results on randomly generated and real matrix completion problems demonstrate that this algorithm is much faster and provides much better recoverability than semidefinite programming solvers such as SDPT3. For example, our algorithm can recover 1000 × 1000 matrices of rank 50 with a relative error of 10−5 in about 3 minutes by sampling only 20 percent of the elements. We know of no other method that achieves as good recoverability. Numerical experiments on online recommendation, DNA microarray data set and image inpainting problems demonstrate the effectiveness of our algorithms. In Chapter 3, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving linearly constrained matrix rank minimization problems are reported. Chapters 4 and 5 considers alternating direction type methods for solving composite convex optimization problems. We present in Chapter 4 alternating linearization algorithms that are based on an alternating direction augmented Lagrangian approach for minimizing the sum of two convex functions. Our basic methods require at most O(1/e) iterations to obtain an e-optimal solution, while our accelerated (i.e., fast) versions require at most O(1/ 3 ) iterations, with little change in the computational effort required at each iteration. For more general problem, i.e., minimizing the sum of K convex functions, we propose multiple-splitting algorithms for solving them. We propose both basic and accelerated algorithms with O(1/e) and O(1/ 3 ) iteration complexity bounds for obtaining an e-optimal solution. To the best of our knowledge, the complexity results presented in these two chapters are the first ones of this type that have been given for splitting and alternating direction type methods. Numerical results on various applications in sparse and low-rank optimization, including compressed sensing, matrix completion, image deblurring, robust principal component analysis, are reported to demonstrate the efficiency of our methods.

This work studies the problem of analyzing and, subsequently, optimizing the stabilization capabilities of a class of controllers for input constrained nonlinear systems. The proposed techniques apply to continuous, state feedback controllers which are defined in a subset of the state space where the time derivative of a known, candidate Control Lyapunov Function (CLF) can be made negative definite under the input constraints. This set is associated with the domain of this class of controllers and depends on the CLF. The analysis problem concerns approximating the domain and is posed via appropriately formulated set containment relationships through the generalized S-procedure with sum of squares (SOS) constraints. The optimization problem is concerned with the adjustment or enlargement of the domain and constitutes a way of controller synthesis. These objectives are pursued via optimizing over the coefficients of polynomial CLFs, through a sequence of semidefinite programming (SDP) problems with SOS constraints. By partitioning the state space based on the structure of the input value set and building upon earlier results on SOS methods, the SDP problems are subject to only convex constraints, rendering thus the proposed techniques computationally viable. The capabilities of the proposed algorithms are demonstrated through numerical examples.

We propose a new homotopy-based conditional gradient method for solving convex optimization problems with a large number of simple conic constraints. Instances of this template naturally appear in semidefinite programming problems arising as convex relaxations of combinatorial optimization problems. Our method is a double-loop algorithm in which the conic constraint is treated via a self-concordant barrier, and the inner loop employs a conditional gradient algorithm to approximate the analytic central path, while the outer loop updates the accuracy imposed on the temporal solution and the homotopy parameter. Our theoretical iteration complexity is competitive when confronted to state-of-the-art semidefinite programming solvers, with the decisive advantage of cheap projection-free subroutines. Preliminary numerical experiments are provided for illustrating the practical performance of the method.

In this paper we consider the general problem of designing the optimal quantum detection for minimum-error quantum state discrimination. Since it is hard to get an explicit solution and we must resort to numerical methods for the general case. We show the problem can be cast into a semidefinite programming(SDP) problem, we use the theory of SDP to solve it. Based on the idea, we propose a method for minimum-error discrimination by formulating the design of optimal quantum detection as a standard SDP problem; furthermore, based on the optimality conditions for SDP, we derive a set of optimality conditions for quantum detection. It is exemplified that this method is simple to implement and can be used to design optimum detection operators for minimum-error discrimination efficiently.

This issue's SIGEST paper, from the SIAM Journal on Optimization (SIOPT), is an excellent example of deep mathematical analysis, involving a range of mathematical techniques, which is having significant practical algorithmic and computational consequences. The title of the paper, “A Sum of Squares Approximation of Nonnegative Polynomials,” by Jean B. Lasserre, combines two concepts that are known to all SIAM readers: sums of squares, the types of functions that arise in many data fitting and other applications; and nonnegative polynomials, polynomials whose values are greater than or equal to zero for all inputs. It is obvious that any polynomial that can be written as a sum of squares is nonnegative. It is far from obvious whether any nonnegative polynomial has a representation as, or arbitrarily close to, a sum of squares of polynomials. In this SIGEST paper, Lasserre shows that this is the case. The nomination from the editors of SIOPT captures the immediate and more general contributions of this paper eloquently, in the context of a series of three papers of which this is the second: “Jean Lasserre, in a series of papers (starting from the first paper in SIOPT Vol. 11, pp. 796–817, and concluding with the third paper in SIOPT Vol. 15, pp. 383–393) by exploiting deep results from real algebraic geometry and theory of moments for multivariate polynomials, showed that the optimal value of a nonconvex polynomial optimization problem (i.e., an optimization problem whose objective function and constraints are specified by multivariate polynomials) can asymptotically be approximated by an increasing sequence of optimal values from increasing tighter semidefinite programming relaxation problems. Even though the result is asymptotic in the most general case, fortuitously, practical experience has revealed that very often, the first few levels of semidefinite programming relaxations typically can achieve the global optimum value. The theoretical connection that Jean Lasserre (and Pablo Parrilo at about the same time from the viewpoint of sum of squares representations of multivariate polynomials) had established between polynomial optimizations with real algebraic geometry and theory of moments has led to exciting developments in combinatorial optimization. In particular, many hard combinatorial problems such as the maximum clique problem of a graph and the quadratic assignment problem are now actively revisited by researchers under the framework of polynomial optimization to derive increasingly tighter bounds.” That is, a consequence of this and related work is the ability to use a sequence of semidefinite programming problems to (nearly) solve much more difficult optimization problems. Semidefinite programming problems are problems with a linear objective function and linear constraints in a matrix variable X, plus the additional constraint that X be positive semidefinite. Since a rich array of efficient algorithms for solving semidefinite programming problems has been developed in recent years, reducing more general problems to a sequence of semidefinite programming problems is an important advance. We hope that readers of SIAM Review will enjoy this glimpse into the sophisticated mathematics behind an important algorithmic innovation in optimization.

In this paper, we address the motion parameter estimation problem using time difference of arrival (TDOA) measurements, where a mobile source starts from an initial position with constant velocity. The problem is formulated as three non-convex constrained weighted least squares (CWLS) problems. Owing to their non-convex nature, local convergence may occur when solving them by an iterative algorithm, implying good estimates are not guaranteed. We propose to solve these CWLS problems by applying semidefinite relaxation (SDR) to generate three different semidefinite programming (SDP) problems, namely, the motion unconstrained semidefinite programming (MU-SDP), motion constrained semidefinite programming (MC-SDP), and motion parameter direct semidefinite programming (MPD-SDP). The MU-SDP, MC-SDP, and MPD-SDP methods are then extended to the scenario of unknown propagation speed (PS), which is common in underwater and underground localizations. Specifically, for this scenario, the two-step MU-SDP, two-step MC-SDP, and two-step MPD-SDP methods are designed to keep the relaxed SDP problems tight by introducing the penalty function. Simulation results show that MU-SDP performs worse than MC-SDP and MPD-SDP owing to the ignorance of the motion equation, and MC-SDP and MPD-SDP are able to reach the Cramér-Rao lower bound (CRLB) accuracy.

Distance metric learning plays an important role in many vision problems. Previous work of quadratic Mahalanobis metric learning usually needs to solve a semidefinite programming (SDP) problem. A standard interior-point SDP solver has a complexity of O(D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6.5</sup> ) (with D the dimension of input data), and can only solve problems up to a few thousand variables. Since the number of variables is D(D + l)/2, this corresponds to a limit around D <; 100. This high complexity hampers the application of metric learning to high-dimensional problems. In this work, we propose a very efficient approach to this metric learning problem. We formulate a Lagrange dual approach which is much simpler to optimize, and we can solve much larger Mahalanobis metric learning problems. Roughly, the proposed approach has a time complexity of O(t · D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) with t ≈ 20 ~ 30 for most problems in our experiments. The proposed algorithm is scalable and easy to implement. Experiments on various datasets show its similar accuracy compared with state-of-the-art. We also demonstrate that this idea may also be able to be applied to other SDP problems such as maximum variance unfolding.

A SemiDefinite Programming (SDP) problem is one of the most central problems in mathematical optimization. SDP provides an effective computation framework for many research fields. Some applications, however, require solving a large-scale SDP whose size exceeds the capacity of a single processor both in terms of computation time and available memory. SDPARA (SemiDefinite Programming Algorithm paRAllel package) [Yamashita et al. 2003b] was designed to solve such large-scale SDPs. Its parallel performance is outstanding for general SDPs in most cases. However, the parallel implementation is less successful for some sparse SDPs obtained from applications such as Polynomial Optimization Problems (POPs) or Sensor Network Localization (SNL) problems, since this version of SDPARA cannot directly handle sparse Schur Complement Matrices (SCMs). In this article we improve SDPARA by focusing on the sparsity of the SCM and we propose a new parallel implementation using the formula-cost-based distribution along with a replacement of the dense Cholesky factorization. We verify numerically that these features are key to solving SDPs with sparse SCMs more quickly on parallel computing systems. The performance is further enhanced by multithreading and the new SDPARA attains considerable scalability in general. It also finds solutions for extremely large-scale SDPs arising from POPs which cannot be obtained by other solvers.

An optimal experimental design represents a structured approach to collecting data with the aim of maximizing the information gleaned. Achieving this requires defining an optimality criterion tailored to the specific model under consideration and the purpose of the investigation. However, it is often observed that a design optimized for one criterion may not perform optimally when applied to another. To mitigate this, one strategy involves employing compound designs. These designs balance multiple criteria to create robust experimental plans that are versatile across different applications. In our study, we systematically tackle the challenge of constructing compound approximate optimal experimental designs using Semidefinite Programming. We focus on discretized design spaces, with the objective function being the geometric or the arithmetic mean of design efficiencies relative to individual criteria. We address two combinations of two criteria: concave-concave (illustrated by DE–optimality) and convex-concave (such as DA–optimality). To handle the latter, we reformulate the problem as a bilevel problem. Here, the outer problem is solved using Surrogate Based Optimization, while the inner problem is addressed with a Semidefinite Programming solver. We demonstrate our formulations using both linear and nonlinear models (for the response) of the Beta class, previously linearized to facilitate analysis and comparison.