Piecewise SOS-convex moment optimization and applications via exact semi-definite programs

Sums of squares based approximation algorithms for MAX-SAT

This issue's SIGEST paper, from the SIAM Journal on Optimization (SIOPT), is an excellent example of deep mathematical analysis, involving a range of mathematical techniques, which is having significant practical algorithmic and computational consequences. The title of the paper, “A Sum of Squares Approximation of Nonnegative Polynomials,” by Jean B. Lasserre, combines two concepts that are known to all SIAM readers: sums of squares, the types of functions that arise in many data fitting and other applications; and nonnegative polynomials, polynomials whose values are greater than or equal to zero for all inputs. It is obvious that any polynomial that can be written as a sum of squares is nonnegative. It is far from obvious whether any nonnegative polynomial has a representation as, or arbitrarily close to, a sum of squares of polynomials. In this SIGEST paper, Lasserre shows that this is the case. The nomination from the editors of SIOPT captures the immediate and more general contributions of this paper eloquently, in the context of a series of three papers of which this is the second: “Jean Lasserre, in a series of papers (starting from the first paper in SIOPT Vol. 11, pp. 796–817, and concluding with the third paper in SIOPT Vol. 15, pp. 383–393) by exploiting deep results from real algebraic geometry and theory of moments for multivariate polynomials, showed that the optimal value of a nonconvex polynomial optimization problem (i.e., an optimization problem whose objective function and constraints are specified by multivariate polynomials) can asymptotically be approximated by an increasing sequence of optimal values from increasing tighter semidefinite programming relaxation problems. Even though the result is asymptotic in the most general case, fortuitously, practical experience has revealed that very often, the first few levels of semidefinite programming relaxations typically can achieve the global optimum value. The theoretical connection that Jean Lasserre (and Pablo Parrilo at about the same time from the viewpoint of sum of squares representations of multivariate polynomials) had established between polynomial optimizations with real algebraic geometry and theory of moments has led to exciting developments in combinatorial optimization. In particular, many hard combinatorial problems such as the maximum clique problem of a graph and the quadratic assignment problem are now actively revisited by researchers under the framework of polynomial optimization to derive increasingly tighter bounds.” That is, a consequence of this and related work is the ability to use a sequence of semidefinite programming problems to (nearly) solve much more difficult optimization problems. Semidefinite programming problems are problems with a linear objective function and linear constraints in a matrix variable X, plus the additional constraint that X be positive semidefinite. Since a rich array of efficient algorithms for solving semidefinite programming problems has been developed in recent years, reducing more general problems to a sequence of semidefinite programming problems is an important advance. We hope that readers of SIAM Review will enjoy this glimpse into the sophisticated mathematics behind an important algorithmic innovation in optimization.

This work studies the problem of analyzing and, subsequently, optimizing the stabilization capabilities of a class of controllers for input constrained nonlinear systems. The proposed techniques apply to continuous, state feedback controllers which are defined in a subset of the state space where the time derivative of a known, candidate Control Lyapunov Function (CLF) can be made negative definite under the input constraints. This set is associated with the domain of this class of controllers and depends on the CLF. The analysis problem concerns approximating the domain and is posed via appropriately formulated set containment relationships through the generalized S-procedure with sum of squares (SOS) constraints. The optimization problem is concerned with the adjustment or enlargement of the domain and constitutes a way of controller synthesis. These objectives are pursued via optimizing over the coefficients of polynomial CLFs, through a sequence of semidefinite programming (SDP) problems with SOS constraints. By partitioning the state space based on the structure of the input value set and building upon earlier results on SOS methods, the SDP problems are subject to only convex constraints, rendering thus the proposed techniques computationally viable. The capabilities of the proposed algorithms are demonstrated through numerical examples.

Solving optimization problems with sparse or low-rank optimal solutions has been an important topic since the recent emergence of compressed sensing and its matrix extensions such as the matrix rank minimization and robust principal component analysis problems. Compressed sensing enables one to recover a signal or image with fewer observations than the “length” of the signal or image, and thus provides potential breakthroughs in applications where data acquisition is costly. However, the potential impact of compressed sensing cannot be realized without efficient optimization algorithms that can handle extremely large-scale and dense data from real applications. Although the convex relaxations of these problems can be reformulated as either linear programming, second-order cone programming or semidefinite programming problems, the standard methods for solving these relaxations are not applicable because the problems are usually of huge size and contain dense data. In this dissertation, we give efficient algorithms for solving these “sparse” optimization problems and analyze the convergence and iteration complexity properties of these algorithms. Chapter 2 presents algorithms for solving the linearly constrained matrix rank minimization problem. The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization. Although the latter can be cast and solved as a semidefinite programming problem, such an approach is computationally expensive when the matrices are large. In Chapter 2, we propose fixed-point and Bregman iterative algorithms for solving the nuclear norm minimization problem and prove convergence of the first of these algorithms. By using a homotopy approach together with an approximate singular value decomposition procedure, we get a very fast, robust and powerful algorithm, which we call FPCA (Fixed Point Continuation with Approximate SVD), that can solve very large matrix rank minimization problems. Our numerical results on randomly generated and real matrix completion problems demonstrate that this algorithm is much faster and provides much better recoverability than semidefinite programming solvers such as SDPT3. For example, our algorithm can recover 1000 × 1000 matrices of rank 50 with a relative error of 10−5 in about 3 minutes by sampling only 20 percent of the elements. We know of no other method that achieves as good recoverability. Numerical experiments on online recommendation, DNA microarray data set and image inpainting problems demonstrate the effectiveness of our algorithms. In Chapter 3, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving linearly constrained matrix rank minimization problems are reported. Chapters 4 and 5 considers alternating direction type methods for solving composite convex optimization problems. We present in Chapter 4 alternating linearization algorithms that are based on an alternating direction augmented Lagrangian approach for minimizing the sum of two convex functions. Our basic methods require at most O(1/e) iterations to obtain an e-optimal solution, while our accelerated (i.e., fast) versions require at most O(1/ 3 ) iterations, with little change in the computational effort required at each iteration. For more general problem, i.e., minimizing the sum of K convex functions, we propose multiple-splitting algorithms for solving them. We propose both basic and accelerated algorithms with O(1/e) and O(1/ 3 ) iteration complexity bounds for obtaining an e-optimal solution. To the best of our knowledge, the complexity results presented in these two chapters are the first ones of this type that have been given for splitting and alternating direction type methods. Numerical results on various applications in sparse and low-rank optimization, including compressed sensing, matrix completion, image deblurring, robust principal component analysis, are reported to demonstrate the efficiency of our methods.

One specific subset of quantum algorithms is Grovers Ordered Search Problem (OSP), the quantum counterpart of the classical binary search algorithm, which utilizes oracle functions to produce a specified value within an ordered database. Classically, the optimal algorithm is known to have a [Formula: see text] complexity; however, Grovers algorithm has been found to have an optimal complexity between the lower bound of [Formula: see text] and the upper bound of [Formula: see text]. We sought to lower the known upper bound of the OSP. With Farhi et al. MITCTP 2815 (1999), arXiv:quant-ph/9901059], we see that the OSP can be resolved into a translational invariant algorithm to create quantum query algorithm restraints. With these restraints, one can find Laurent polynomials for various [Formula: see text] — queries — and [Formula: see text] — database sizes — thus finding larger recursive sets to solve the OSP and effectively reducing the upper bound. These polynomials are found to be convex functions, allowing one to make use of convex optimization to find an improvement on the known bounds. According to Childs et al. [Phys. Rev. A 75 (2007) 032335], semidefinite programming, a subset of convex optimization, can solve the particular problem represented by the constraints. We were able to implement a program abiding to their formulation of a semidefinite program (SDP), leading us to find that it takes an immense amount of storage and time to compute. To combat this setback, we then formulated an approach to improve results of the SDP using matrix sparsity. Through the development of this approach, along with an implementation of a rudimentary solver, we demonstrate how matrix sparsity reduces the amount of time and storage required to compute the SDP — overall ensuring further improvements will likely be made to reach the theorized lower bound.

Sum of squares (SOS) optimization has been a powerful and influential addition to the theory of optimization in the past decade. Its reliance on relatively large-scale semidefinite programming, however, has seriously challenged its ability to scale in many practical applications. In this paper, we introduce DSOS and SDSOS optimization as more tractable alternatives to sum of squares optimization that rely instead on linear programming and second order cone programming. These are optimization problems over certain subsets of sum of squares polynomials and positive semidefinite matrices and can be of potential interest in general applications of semidefinite programming where scalability is a limitation.

Representation of a given nonnegative multivariate polynomial in terms of a sum of squares of polynomials has become an essential subject in recent developments of sums of squares optimization and semidefinite programming (SDP) relaxation of polynomial optimization problems. We discuss effective methods to obtain a simpler representation of a “sparse” polynomial as a sum of squares of sparse polynomials by eliminating redundancy.

Although sum of squares programming has been used extensively over the past decade for the stability analysis of nonlinear systems, several fundamental questions remain unanswered. In this paper, we show that exponential stability of a polynomial vector field on a bounded set implies the existence of a Lyapunov function which is a sum of squares of polynomials. In particular, the main result states that if a system is exponentially stable on a bounded nonempty set, then there exists a sum of squares Lyapunov function which is exponentially decreasing on that bounded set. Furthermore, we derive a bound on the degree of this converse Lyapunov function as a function of the continuity and stability properties of the vector field. The proof is constructive and uses the Picard iteration. Our result implies that semidefinite programming can be used to answer the question of stability of a polynomial vector field with a bound on complexity.

Moment-SOS relaxation of the medium term hydrothermal dispatch problem

Semialgebraic computations, i.e., the manipulation of sets and logical conditions defined by polynomial inequalities in real variables, are an essential ”primitive” in the analysis and design of hybrid dynamical systems. Fundamental tasks such as reachability analysis, abstraction verification, and the computation of stability and performance certificates, all use these operations extensively, and can quickly become the computational bottleneck in the design process. Although there is a well-developed body of both basic theory and algorithms for these tasks, the practical performance of most available methods is still far from being satisfactory on real-world problems. While there are several possible causes for this (besides the NP-hardness of the task), a sensible explanation lies in the purely algebraic nature of the usual methods, as well as the insistence on exact (as opposed to approximate or ”relaxed”) solutions. For these reasons, there is a strong interest in the development of efficient techniques for (perhaps restricted) classes of semialgebraic problems. In this talk we review the basic elements and present several new results on the SOS approach to semialgebraic computations, that combines symbolic and numerical techniques from real algebra and convex optimization. Its main defining feature is the use and computation of sum of squares (SOS) decompositions for multivariate polynomials via semidefinite programming. These are extended, using the Positivstellensatz, to structured infeasibility certificates for polynomial equations and inequalities. The developed techniques unify and generalize many well-known existing methods. In particular, we will discuss semialgebraic problems with at most two quantifier alternations (i.e., classical polynomial optimization problems as well as the related games and minimax problems). As an example, we will solve in detail a class of zero-sum two-person games with an infinite number of pure strategies, where the payoff function is a polynomial expression of the actions of the players. Although particular emphasis will be given to the hybrid systems viewpoint, the basic ideas and algorithms, as well as these recent extensions, will be illustrated with examples drawn from a broad range of related domains, including dynamical systems and geometric theorem proving.

We begin a systematic study of positivity and moment problems in an equivariant setting. Given a reductive groupGGoverR\mathbb {R}acting on an affineR\mathbb {R}-varietyVV, we consider the induced dual action on the coordinate ringR[V]\mathbb {R}[V]and on the linear dual space ofR[V]\mathbb {R}[V]. In this setting, given an invariant closed semialgebraic subsetKKofV(R)V(\mathbb R), we study the problem of representation of invariant nonnegative polynomials onKKby invariant sums of squares, and the closely related problem of representation of invariant linear functionals onR[V]\mathbb {R}[V]by invariant measures supported onKK. To this end, we analyse the relation between quadratic modules ofR[V]\mathbb {R}[V]and associated quadratic modules of the (finitely generated) subringR[V]G\mathbb {R}[V]^Gof invariant polynomials. We apply our results to investigate the finite solvability of an equivariant version of the multidimensionalKK-moment problem. Most of our results are specific to the case where the groupG(R)G(\mathbb {R})is compact.

This work is concerned with the proof-complexity of certifying that optimization problems do not have good solutions. Specifically we consider bounded-degree Sum of Squares (SOS) proofs, a powerful algebraic proof system introduced in 1999 by Grigoriev and Vorobjov. Work of Shor, Lasserre, and Parrilo shows that this proof is automatizable using semidefinite programming (SDP), meaning that any n-variable degree-d proof can be found in time nO(d). Furthermore, the SDP is dual to the well-known Lasserre SDP hierarchy, meaning that the d/2-round Lasserre of an optimization problem is equal to the best bound provable using a degree-d SOS proof. These ideas were exploited in a recent paper by Barak et al. (STOC 2012) which shows that the known hard for the Unique-Games problem are in fact optimally solved by a constant level of the Lasserre SDP hierarchy.We continue the study of the power of SOS proofs in the context of difficult optimization problems. In particular, we show that the Balanced-Separator integrality gap instances proposed by Devanur et al. can have their optimal value certified by a degree-4 SOS proof. The key ingredient is an SOS proof of the KKL Theorem. We also investigate the extent to which the Khot--Vishnoi Max-Cut integrality gap instances can have their optimum value certified by an SOS proof. We show they can be certified to within a factor .952 (> .878) using a constant-degree proof. These investigations also raise an interesting mathematical question: is there a constant-degree SOS proof of the Central Limit Theorem?

We demonstrate applications of algebraic techniques that optimize and certify polynomial inequalities to problems of interest in the operations research and transportation engineering communities. Three problems are considered: (1) wireless coverage of targeted geographical regions with guaranteed signal quality and minimum transmission power, (2) computing real-time certificates of collision avoidance for a simple model of an unmanned vehicle (UV) navigating through a cluttered environment, and (3) designing a nonlinear hovering controller for a quadrotor UV, which has recently been used for load transportation. On our smaller-scale applications, we apply the sum of squares (SOS) relaxation and solve the underlying problems with semidefinite programming. On the larger-scale or real-time applications, we use our recently introduced “SDSOS Optimization” techniques which result in second order cone programs. To the best of our knowledge, this is the first study of real-time applications of sum of squares techniques in optimization and control. No knowledge in dynamics and control is assumed from the reader.

This paper deals with stability analysis of hybrid systems. Such systems are characterized by a combination of continuous dynamics and logic based switching between discrete modes. Lyapunov theory is a well known methodology for the stability analysis of linear and nonlinear systems in control system literature. Construction of Lyapunov functions for hybrid systems is generally a difficult task, but once these functions are defined, stabilization of the system is straight-forward. The sum of squares (SOS) decomposition and semidefinite programming has also provided an efficient methodology for analysis of nonlinear systems. The computational method used in this paper relies on the SOS decomposition of multivariate polynomials. By using SOS, we construct a (some) Lyapunov function(s) for the hybrid system. The reduction techniques provide numerical solution of large-scale instances; otherwise they will be practically unsolvable. The introduced method can be used for hybrid systems with linear or nonlinear vector fields. Some examples are given to demonstrate the capabilities of the proposed approach.

We study the optimal pricing for revenue maximization over social networks in the presence of positive network externalities. In our model, the value of a digital good for a buyer is a function of the set of buyers who have already bought the item. In this setting, a decision to buy an item depends on its price and also on the set of other buyers that have already owned that item. The revenue maximization problem in the context of social networks has been studied by Hartline, Mirrokni, and Sundararajan [4], following the previous line of research on optimal viral marketing over social networks [5,6,7].We consider the Bayesian setting in which there are some prior knowledge of the probability distribution on the valuations of buyers. In particular, we study two iterative pricing models in which a seller iteratively posts a new price for a digital good (visible to all buyers). In one model, re-pricing of the items are only allowed at a limited rate. For this case, we give a FPTAS for the optimal pricing strategy in the general case. In the second model, we allow very frequent re-pricing of the items. We show that the revenue maximization problem in this case is inapproximable even for simple deterministic valuation functions. In the light of this hardness result, we present constant and logarithmic approximation algorithms when the individual distributions are identical.