Learning to Identify Sources of Network Diffusion

We consider an approach for solving strictly convex quadratic programs (QPs) with general linear inequalities by the alternating direction method of multipliers (ADMM). In particular, we focus on the application of ADMM to the QPs of constrained Model Predictive Control (MPC). After introducing our ADMM iteration, we provide a proof of convergence closely related to the theory of maximal monotone operators. The proof relies on a general measure to monitor the rate of convergence and hence to characterize the optimal step size for the iterations. We show that the identified measure converges at a Q-linear rate while the iterates converge at a 2-step Q-linear rate. This result allows us to relax some of the existing assumptions in optimal step size selection, that currently limit the applicability to the QPs of MPC. The results are validated through a large public benchmark set of QPs of MPC for controlling a four tank process.

Complex ADMM-Net, a complex-valued neural network architecture inspired by the alternating direction method of multipliers (ADMM), is designed for interference removal in super-resolution stepped frequency radar angle-range-doppler imaging. Tailored to an uncooperative scenario wherein a MIMO radar shares spectrum with communications, the ADMM-Net recovers the radar image---which is assumed to be sparse---and simultaneously removes the communication interference, which is modeled as sparse in the frequency domain owing to spectrum underutilization. The scenario motivates an $\ell_1$-minimization problem whose ADMM iteration, in turn, undergirds the neural network design, yielding a set of generalized ADMM iterations that have learnable hyperparameters and operations. To train the network we use random data generated according to the radar and communication signal models. In numerical experiments ADMM-Net exhibits markedly lower error and computational cost than ADMM and CVX.

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

The alternating direction method of multipliers (ADMM) with over-relaxation and the accelerated ADMM are two effective methods to improve the convergence rate of ADMM. In this paper, an efficient linear programming (LP) decoder based on accelerated ADMM for low-density parity-check (LDPC) codes is proposed. Then it is shown that the ADMM update steps for decoding LDPC codes using the accelerated ADMM and the over-relaxed ADMM are quite similar. Therefore, it is impossible to obtain a more efficient ADMM decoder by combining the two improved ADMM methods. Simulation results also validate that the average number of iterations for LDPC decoding using the accelerated ADMM and the over-relaxed ADMM are close to each other.

In model predictive control, the control action is found at each sampling time by solving an online optimization problem. Computationally, this step is very demanding, especially if compared to the evaluation of traditional control laws. This has limited the application of model predictive control to systems with slow dynamics for many years. Recently, several methods have been proposed in the literature which promise a substantial reduction of the computation time by either running the computation in parallel (distributed model predictive control) or exploiting the problem structure (fast model predictive control). A combination of these methods has not yet been considered in the literature. To achieve this goal, different optimization techniques need to be employed at once. The order of how these methods are applied matters. This paper considers fast distributed model predictive control combining the alternating direction method of multipliers (ADMM), the interior point method (IPM) and the Riccati iteration for a particular class of multi-agent systems for which the order of the methods can be arbitrarily changed. This leads to two different solver schemes where a trade-off arises between computation time and number of communications required to reach consensus. A simplified problem involving the formation control of a fleet of vehicles is considered at the end.

Alternating Direction Method of Multiplier (ADMM) has been a popular algorithmic framework for separable optimization problems with linear constraints. For numerical ADMM fail to exploit the particular structure of the problem at hand nor the input data information, leveraging task-specific modules (e.g., neural networks and other data-driven architectures) to extend ADMM is a significant but challenging task. This work focuses on designing a flexible algorithmic framework to incorporate various task-specific modules (with no additional constraints) to improve the performance of ADMM in real-world applications. Specifically, we propose Guidance from Optimality (GO), a new customization strategy, to embed task-specific modules into ADMM (GO-ADMM). By introducing an optimality-based criterion to guide the propagation, GO-ADMM establishes an updating scheme agnostic to the choice of additional modules. The existing task-specific methods just plug their task-specific modules into the numerical iterations in a straightforward manner. Even with some restrictive constraints on the plug-in modules, they can only obtain some relatively weaker convergence properties for the resulted ADMM iterations. Fortunately, without any restrictions on the embedded modules, we prove the convergence of GO-ADMM regarding objective values and constraint violations, and derive the worst-case convergence rate measured by iteration complexity. Extensive experiments are conducted to verify the theoretical results and demonstrate the efficiency of GO-ADMM.

Reversible data hiding (RDH) is desirable in applications where both the hidden message and the cover image need to be recovered without loss. Among many RDH approaches is prediction-error expansion (PEE), containing two steps: i) prediction of a target pixel value, and ii) embedding according to the value of the prediction error. In general, higher prediction performance leads to larger embedding capacity and/or lower signal distortion. Leveraging on recent advances in graph signal processing (GSP), we pose pixel prediction as a graph-signal restoration problem, and design non-local graph-based prediction schemes where the appropriate edge weights of the underlying graph are computed using a similar patch searched in a semi-local neighborhood. Specifically, for each candidate patch, we first examine eigenvalues of its structure tensor to estimate its local smoothness. If sufficiently smooth, we pose a maximum a posteriori (MAP) problem using either a quadratic Laplacian regularizer or a graph total variation (GTV) term as signal prior. While the MAP problem using the first prior has a closed-form solution, we design an efficient algorithm for the second prior using alternating direction method of multipliers (ADMM) with nested proximal gradient descent. Finally, hidden message will be embedded into the image according to the resulting prediction errors. Experimental results show that with better quality GSP-based prediction, at low capacity the visual quality of the embedded image exceeds state-of-the-art methods noticeably.

Purpose:To propose a new parallel‐friendly optimization algorithm, based on alternating direction method of multipliers (ADMM), for solving optimization problems emerging in treatment planning and imagingMethods:ADMM, emerging as a powerful tool for distributed optimization, is employed here for IMRT treatment planning. We modify the existing ADMM by integrating that with two optimization techniques known as Barzilai‐Borwein gradient method and line search. We apply original and integrated ADMM, with various penalty parameter values, on three IMRT treatment planning cases and we compare their performance. Each algorithm terminates once it found a solution within the relative error of 1E‐4 from the optimal solution. We also compare the performance of original and integrated ADMM against the commercial optimization software CPLEX.Results:Our experiments on three different cancer sites (prostate, head and neck, and GYN) indicate that integrated ADMM is much faster than original ADMM (5.8 times faster on average). Moreover, while the performance of original ADMM heavily depends on the selected penalty parameter, integrated ADMM performs very robust against the penalty parameter. Compared to the commercial software CPLEX, integrated ADMM turns out to be around 4–7 times faster.Conclusion:Integrated ADMM is a very efficient optimization algorithm to cope with the large‐scale optimization problems. It is a parallel‐friendly optimization algorithm and provides an idea platform for cloud‐based treatment planning and imaging.

The alternating direction method of multipliers (ADMM) is applied to constrained quadratic optimization of intensity-modulated radiation therapy (IMRT) planning. The proximal operator for the ADMM algorithm is developed, and the algorithm is applied to different clinical treatment sites (liver, prostate, and head & neck). The results are compared with the solutions obtained by several other commercial and non-commercial optimizers. The ADMM solver is about one to two orders of magnitude faster than the other solvers, moreover, it requires less computer memory.

Distributed model predictive control (DMPC) is a widely used approach especially for large-scale systems like transportation, traffic control or power distribution systems where solving the MPC problem in a centralized manner is not desirable or even not possible due to the size of the problem, maintenance issues, cabling burden, etc. Distributed optimization methods, which are being employed to solve the DMPC problems, normally suffer from having an enormous number of communications to reach a consensus between the local controllers. Therefore trying to find distributed optimization schemes that can realize an agreement among the local controllers with a lower communication load deserves attention. The alternating direction method of multipliers (ADMM) is a well known method to solve consensus problems. Since the local problems in ADMM can be handled with second-order methods, the convergence will be reached with a lower number of iterations in comparison to first-order methods, e.g. the fast gradient method. In this paper, ADMM is formulated and applied to a DMPC problem first in the primal domain. The main contribution of this paper is, however, the application of ADMM to the dual of the DMPC problem, introducing an augmented connection graph, which shows considerable improvement in convergence speed and therefore requires a lower number of communications in comparison with primal ADMM.

We focus on the convergence rate of the alternating direction method of multipliers (ADMM) in a complex domain. First, the complex form of variational inequality (VI) is established by using the Wirtinger calculus technique. Second, the \(O(1/K)\) convergence rate of the ADMM in a complex domain is provided. Third, the ADMM in a complex domain is applied to the least absolute shrinkage and selectionator operator (LASSO). Finally, numerical simulations are provided to show that ADMM in a complex domain has the \(O(1/K)\) convergence rate and that it has certain advantages compared with the ADMM in a real domain. doi:10.1017/S1446181118000184

Decoding by alternating direction method of multipliers (ADMM) is a promising linear programming decoder for low-density parity-check (LDPC) codes. In this paper, we propose a two-step scheme to lower the error floor of LDPC codes with ADMM penalized decoder. For the undetected errors that cannot be avoided at the decoder side, we modify the code structure slightly to eliminate low-weight code words. For the detected errors induced by small error-prone structures, we propose a post-processing method for the ADMM penalized decoder. Simulation results show that the error floor can be reduced significantly over three illustrated LDPC codes by the proposed two-step scheme.

The alternating direction method of multipliers (ADMM) is an algorithm for solving large-scale data optimization problems in machine learning. In order to reduce the communication delay in a distributed environment, asynchronous distributed ADMM (AD-ADMM) was proposed. However, due to the unbalance process arrival pattern existing in the multiprocessor cluster, the communication of the star structure used in AD-ADMM is inefficient. Moreover, the load in the entire cluster is unbalanced, resulting in a decrease of the data processing capacity. This paper proposes a hierarchical parameter server communication structure (HPS) and an asynchronous distributed ADMM (HAD-ADMM). The algorithm mitigates the unbalanced arrival problem through process grouping and scattered updating global variable, which basically achieves load balancing. Experiments show that the HAD-ADMM is highly efficient in a large-scale distributed environment and has no significant impact on convergence.

In this work, we leverage the alternating direction method of multipliers (ADMM) framework to solve task assignment for a multi-robot team. While ADMM is a well-established method, it has yet to be utilized in multi-robot problems as the standard formulation requires a centralized update step, a paradigm that conflicts with decentralization as a means of robustness. Therefore, we describe the formulation of separable optimizations in order to produce decentralized ADMM algorithms. Here, our aim is to provide an additional tool for solving cooperative team-based problems in robotics. For the decentralized algorithms, we discuss the conditions for convergence to the optimal centralized solution. We present simulation results for task assignment to demonstrate the benefits of ADMM compared to state-of-the-art methods.KeywordsADMMTask assignmentConsensus methodsCooperative teams

For low-density parity-check (LDPC) codes decoding by alternating direction method of multipliers (ADMM), the layered scheduling sequentially updates the messages of check nodes one by one. Though the layered scheduling can speed up the convergence rates, it may limit the throughput when implementing the ADMM decoder with multi-core systems due to its serial style. To circumvent this problem, a group-based layered scheduling is proposed by updating a group of check node messages at one time. Extensive simulation results for the proposed scheme over two typical LDPC codes with the ADMM penalized decoding algorithm are provided.