Chapter 2: Products and Sparse Problems

This paper introduces tensor methods for solving large, sparse nonlinear least squares problems where the Jacobian either is analytically available or is computed by finite difference approximations. Tensor methods have been shown to have very good computational performance for small to medium-sized, dense nonlinear least squares problems. In this paper we consider the application of tensor methods to large, sparse nonlinear least squares problems where a sparse factorization of the Jacobian can be stored. This involves an entirely new way of solving the tensor model that is efficient for sparse problems. A number of interesting linear algebraic implementation issues are addressed. The test results of the tensor method applied to a set of sparse nonlinear least squares problems compared with those of the standard Gauss--Newton method reveal that the tensor method is significantly more robust and efficient than the standard Gauss--Newton method.

A Cluster-Based Competitive Particle Swarm Optimizer with a Sparse Truncation Operator for Multi-Objective Optimization

The solutions of sparse eigenvalue problems and linear systems constitute one of the key computational kernels in the discretization of partial differential equations for the modeling of linear accelerators. The computational challenges faced by existing techniques for solving those sparse eigenvalue problems and linear systems call for continuing research to improve on the algorithms so that ever increasing problem size as required by the physics application can be tackled. Under the support of this award, the filter algorithm for solving large sparse eigenvalue problems was developed at Stanford to address the computational difficulties in the previous methods with the goal to enable accelerator simulations on then the world largest unclassified supercomputer at NERSC for this class of problems. Specifically, a new method, the Hemitian skew-Hemitian splitting method, was proposed and researched as an improved method for solving linear systems with non-Hermitian positive definite and semidefinite matrices.

Group sparse vectors are the generalization of unstructured sparse vectors, with zero and non-zero elements occurring in groups. In group sparse signal recovery problems, we are interested in finding the signal with the smallest number of active groups that satisfy our observations. Because this problem has exponential complexity, a convex relaxation is typically used, which minimizes the sum of the groups' second norm that satisfies the observations. In this paper, we provide a set of deterministic necessary and sufficient conditions that the sensing matrix should satisfy for equivalence between the l0 solution of a structured group sparse problem and its convex relaxation. These conditions are generalization of the Range Space Property that has been previously proposed for equivalence between the l0- and the l1- norms in non-structured sparse recovery problems. We also provide a sufficient condition for a unique solution of the relaxed convex problem.

An enhanced competitive swarm optimizer with strongly robust sparse operator for large-scale sparse multi-objective optimization problem

Parallel solution of sparse linear least squares problems on distributed-memory multiprocessors

This paper studies sparse linear regression analysis with outliers in the responses.A parameter vector for modeling outliers is added to the standard linear regression model and then the sparse estimation problem for both coefficients and outliers is considered.The 1 penalty is imposed for the coefficients, while various penalties including redescending type penalties are for the outliers.To solve the sparse estimation problem, we introduce an optimization algorithm.Under some conditions, we show the algorithmic and statistical convergence property for the coefficients obtained by the algorithm.Moreover, it is shown that the algorithm can recover the true support of the coefficients with probability going to one.

Massively parallel computation (MPC) algorithms for graph problems have witnessed a resurgence of interest in recent years. Despite major progress for numerous graph problems however, the complexity of the sparse graph connectivity problem in this model has remained elusive: While classical logarithmic-round PRAM algorithms for finding connected components in any n-vertex graph have been known for more than three decades (and imply the same bounds for MPC model), no o(log n)-round MPC algorithms are known for this task with truly sublinear in n memory per machine (which is the only interesting regime for sparse graphs with O(n) edges). It is conjectured that an o(log n)-round algorithm for connectivity on general sparse graphs with n1-Ω (1) per-machine memory may not exist, a conjecture that also forms the basis for multiple conditional hardness results on the round complexity of other problems in the MPC model.We take an opportunistic approach towards the sparse graph connectivity problem by designing an algorithm with improved performance in terms of the connectivity structure of the input graph. Formally, we design an MPC algorithm that finds all connected components with spectral gap at least λ in a graph in O(log log n + log(1/λ)) MPC rounds and nδ memory per machine for any constant δ ∈ (0,1). While this algorithm still requires Θ(log n) rounds in the worst-case, it achieves an exponential round reduction on well-connected'' components with λ ≥ 1/polylog(n) using only nδ memory per machine and l(n) total memory, and still operates in o(log n)l rounds even when λ = 1/no(1).En-route to our main result, we design a new distributed data structure for performing independent random walks from all vertices simultaneously, as well as a new leader-election algorithm with exponentially faster round complexity on random graphs.

We present a framework for optimizing sparse quadratic assignment problems. We propose an iterative algorithm that dynamically generates the quadratic part of the assignment problem and, thus, solves a sparsified linearization of the original problem in every iteration. This procedure results in a hierarchy of lower bounds and, in addition, provides heuristic primal solutions in every iteration. This framework was motivated by the task of the French government to design the French keyboard standard, which included solving sparse quadratic assignment problems with over 100 special characters; a size where many commonly used approaches fail. The design of a new standard often involves conflicting opinions of multiple stakeholders in a committee. Hence, there is no agreement on a single well-defined objective function that can be used for an extensive one-shot optimization. Instead, the process is highly interactive and demands rapid prototyping, e.g., quick primal solutions, on-the-fly evaluation of manual changes, and prompt assessments of solution quality. Particularly concerning the latter aspect, our algorithm is able to provide high-quality lower bounds for these problems in several minutes.

The Fujitsu Digital Annealer (DA) is designed to solve fully connected quadratic unconstrained binary optimization (QUBO) problems. It is implemented on application-specific CMOS hardware and currently solves problems of up to 1024 variables. The DA's algorithm is currently based on simulated annealing; however, it differs from it in its utilization of an efficient parallel-trial scheme and a dynamic escape mechanism. In addition, the DA exploits the massive parallelization that custom application-specific CMOS hardware allows. We compare the performance of the DA to simulated annealing and parallel tempering with isoenergetic cluster moves on two-dimensional and fully connected spin-glass problems with bimodal and Gaussian couplings. These represent the respective limits of sparse versus dense problems, as well as high-degeneracy versus low-degeneracy problems. Our results show that the DA currently exhibits a time-to-solution speedup of roughly two orders of magnitude for fully connected spin-glass problems with bimodal or Gaussian couplings, over the single-core implementations of simulated annealing and parallel tempering Monte Carlo used in this study. The DA does not appear to exhibit a speedup for sparse two-dimensional spin-glass problems, which we explain on theoretical grounds. We also benchmarked an early implementation of the Parallel Tempering DA. Our results suggest an improved scaling over the other algorithms for fully connected problems of average difficulty with bimodal disorder. The next generation of the DA is expected to be able to solve fully connected problems up to 8192 variables in size. This would enable the study of fundamental physics problems and industrial applications that were previously inaccessible using standard computing hardware or special-purpose quantum annealing machines.

The objective of this work is to quantify the reconstruction error in sparse inverse problems with measures and stochastic noise, motivated by optimal sensor placement. To be useful in this context, the error quantities must be explicit in the sensor configuration and robust with respect to the source, yet relatively easy to compute in practice, compared to a direct evaluation of the error by a large number of samples. In particular, we consider the identification of a measure consisting of an unknown linear combination of point sources from a finite number of measurements contaminated by Gaussian noise. The statistical framework for recovery relies on two main ingredients: first, a convex but non-smooth variational Tikhonov point estimator over the space of Radon measures and, second, a suitable mean-squared error based on its Hellinger–Kantorovich distance to the ground truth. To quantify the error, we employ a non-degenerate source condition as well as careful linearization arguments to derive a computable upper bound. This leads to asymptotically sharp error estimates in expectation that are explicit in the sensor configuration. Thus they can be used to estimate the expected reconstruction error for a given sensor configuration and guide the placement of sensors in sparse inverse problems.

We aim to compute lifted stationary points of a sparse optimization problem ( $$P_{0}$$ ) with complementarity constraints. We define a continuous relaxation problem ( $$R_{\nu }$$ ) that has the same global minimizers and optimal value with problem ( $$P_{0}$$ ). Problem ( $$R_{\nu }$$ ) is a mathematical program with complementarity constraints (MPCC) and a difference-of-convex objective function. We define MPCC lifted-stationarity of ( $$R_{\nu }$$ ) and show that it is weaker than directional stationarity, but stronger than Clarke stationarity for local optimality. Moreover, we propose an approximation method to solve ( $$R_{\nu }$$ ) and an augmented Lagrangian method to solve its subproblem ( $$R_{\nu ,\sigma }$$ ), which relaxes the equality constraint in ( $$R_{\nu }$$ ) with a tolerance $$\sigma $$ . We prove the convergence of our algorithm to an MPCC lifted-stationary point of problem ( $$R_{\nu }$$ ) and use a sparse optimization problem with vertical linear complementarity constraints to demonstrate the efficiency of our algorithm on finding sparse solutions in practice.

In real-world applications, there exist a lot of multiobjective optimization problems whose Pareto-optimal solutions are sparse, that is, most variables of these solutions are 0. Generally, many sparse multiobjective optimization problems (SMOPs) contain a large number of variables, which pose grand challenges for evolutionary algorithms to find the optimal solutions efficiently. To address the curse of dimensionality, this article proposes an evolutionary algorithm for solving large-scale SMOPs, which aims to mine the sparse distribution of the Pareto-optimal solutions and, thus, considerably reduces the search space. More specifically, the proposed algorithm suggests an evolutionary pattern mining approach to detect the maximum and minimum candidate sets of the nonzero variables in the Pareto-optimal solutions, and uses them to limit the dimensions in generating offspring solutions. For further performance enhancement, a binary crossover operator and a binary mutation operator are designed to ensure the sparsity of solutions. According to the results on eight benchmark problems and four real-world problems, the proposed algorithm is superior over existing evolutionary algorithms in solving large-scale SMOPs.

Adaptive Neighbors Graph Learning for Large-Scale Data Clustering using Vector Quantization and Self-Regularization

Chapter 13 - Robust and Scalable Shape Prior Modeling via Sparse Representation and Dictionary Learning