A randomized dual consensus ADMM method for multi-agent distributed optimization

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

Distributed model predictive control (DMPC) is a widely used approach especially for large-scale systems like transportation, traffic control or power distribution systems where solving the MPC problem in a centralized manner is not desirable or even not possible due to the size of the problem, maintenance issues, cabling burden, etc. Distributed optimization methods, which are being employed to solve the DMPC problems, normally suffer from having an enormous number of communications to reach a consensus between the local controllers. Therefore trying to find distributed optimization schemes that can realize an agreement among the local controllers with a lower communication load deserves attention. The alternating direction method of multipliers (ADMM) is a well known method to solve consensus problems. Since the local problems in ADMM can be handled with second-order methods, the convergence will be reached with a lower number of iterations in comparison to first-order methods, e.g. the fast gradient method. In this paper, ADMM is formulated and applied to a DMPC problem first in the primal domain. The main contribution of this paper is, however, the application of ADMM to the dual of the DMPC problem, introducing an augmented connection graph, which shows considerable improvement in convergence speed and therefore requires a lower number of communications in comparison with primal ADMM.

The super-resolution algorithms based on deep learning can effectively increase optical remote sensing image (ORSI) details for further analysis tasks. Deep unfolding methods have been studied in recent years to bridge the gap between optimization-based and learning-based methods, which can take good advantage of the prior knowledge. However, these unfolding methods usually ignore the utilization of intermediate network features between different iteration stages, thereby limiting the performance of super-resolution results. We propose a multi-source information fusion network (MSFNet) for ORSI super-resolution to address this problem. We mainly consider three strategies to enhance the image super-resolution performance, including feature extraction strategy, information fusion strategy, and the structure of the unfolding network. Firstly, image information of various scales is helpful for mining potential features of images for image super-resolution. Therefore, we introduce multi-scale implicit constraints to the objective function. Secondly, we unfold the optimization process into a neural network by alternating direction method of multipliers (ADMM). The network framework follows the ADMM method's iteration process. This unfolding strategy can effectively utilize the prior information for image reconstruction. Thirdly, we propose a row-column decoupling Transformer module based on this unfolding framework for feature fusion. Specifically, the row Transformer block completes the feature fusion of various scales, and the column Transformer block completes the feature fusion of various channels. The fused features are transmitted to the next iteration stage for feature enhancement. And this fusion strategy enables the network to focus on the global information of the features. We perform experiments on three remote sensing image datasets to fully demonstrate the algorithm's effectiveness. Experiment results show that the proposed algorithm can achieve better image reconstruction performance.

In microgrids, the volatility and uncertainty of renewable energy make it challenging to achieve optimal performance using hierarchical control strategies with time-scale separation for economic dispatch and real-time control. To address this issue and enhance transient optimization, a distributed economic model predictive control (EMPC) strategy based on the alternating direction method of multipliers (ADMM) is proposed in this paper. To reduce computational complexity and communication overhead in distributed control, the ADMM method is employed to solve the distributed EMPC problem in parallel. Using Lyapunov-based analysis, the study establishes the system's asymptotic stability. Moreover, it demonstrates that the EMPC strategy achieves an average performance equivalent to-or even exceeding-that of the optimal steady-state. Simulations validate the proposed algorithm's effectiveness in economic dispatch and frequency restoration, while demonstrating its advantages in minimizing computational load and communication requirements.

Weighted matrix based distributed optimization method for economic dispatch of microgrids via multi-step gradient descent

Recently, the alternating direction method of multipliers (ADMM) has been used effectively to solve the multi-agent unconstrained optimization problems, where the objective function is the sum of privately known local objective functions of agents. In this paper, first, with the help of the edge-node incidence matrix, an unconstrained optimization problem is transformed into an equivalent optimization problem with only equality constraint and, thus, can be dealt with the ADMM conveniently. Second, a novel distributed inexact consensus ADMM is proposed to enable the agents to reach consensus on the optimal solution of the optimization problem. At the same time, the analysis of the linear convergence of the proposed algorithm is also provided under some mild conditions. Finally, some simulation results are presented to demonstrate the better effectiveness of the proposed algorithm than the standard consensus-based ADMM algorithm.

The alternating direction method of multipliers (ADMM) with over-relaxation and the accelerated ADMM are two effective methods to improve the convergence rate of ADMM. In this paper, an efficient linear programming (LP) decoder based on accelerated ADMM for low-density parity-check (LDPC) codes is proposed. Then it is shown that the ADMM update steps for decoding LDPC codes using the accelerated ADMM and the over-relaxed ADMM are quite similar. Therefore, it is impossible to obtain a more efficient ADMM decoder by combining the two improved ADMM methods. Simulation results also validate that the average number of iterations for LDPC decoding using the accelerated ADMM and the over-relaxed ADMM are close to each other.

In this work, we consider the asynchronous distributed optimization problem in which each node has its own convex cost function and can communicate directly only with its neighbors, as determined by a directed communication topology (directed graph or digraph). First, we reformulate the optimization problem so that Alternating Direction Method of Multipliers (ADMM) can be utilized. Then, we propose an algorithm, herein called Asynchronous Approximate Distributed Alternating Direction Method of Multipliers (AsyAD-ADMM), using finite-time asynchronous approximate ratio consensus, to solve the multi-node convex optimization problem, in which every node performs iterative computations and exchanges information with its neighbors asynchronously. More specifically, at every iteration of AsyAD-ADMM, each node solves a local convex optimization problem for the one of the primal variables and utilizes a finite-time asynchronous approximate consensus protocol to obtain the value of the other variable which is close to the optimal value, since the cost function for the second primal variable is not decomposable. If the individual cost functions are convex, but not-necessarily differentiable, the proposed algorithm converges at a rate of O(1/k), where k is the iteration counter. The efficacy of AsyAD-ADMM is exemplified via a proof-of-concept distributed least square optimization problem with different performance-influencing factors investigated.

This paper presents two strategies to speed up the alternating direction method of multipliers (ADMM) for distributed data. In the first method, inspired by stochastic gradient descent, each machine uses only a subset of its data at the first few iterations, speeding up those iterations. A key result is in proving that despite this approximation, our method enjoys the same convergence rate in terms of the number of iterations as the standard ADMM, and hence is faster overall. The second method also follows the idea of sampling a subset of the data to update the model before the communication of each round. It converts an objective to the approximated dual form and performs ADMM on the dual. The method turns out to be a distributed variant of the recently proposed SDCA-ADMM. Yet, compared to the straightforward distributed implementation of SDCA-ADMM, the proposed method enjoys less frequent communication between machines, better memory usage, and lighter computational demand. Experiments demonstrate the effectiveness of our two strategies.

Data collection plays an important role in wireless sensor networks. Recovery of spatio-temporal data from incomplete sensing data is vital to the network lifetime. Many works have utilized the spatial and temporal correlations to achieve satisfactory data recovery results. However, these methods introduce large computational overhead at the fusion center. In this paper, we develop an ADMM-Net framework for correlated spatio-temporal data recovery. Both the spatial correlation and temporal correlation of sensing data are considered in a convex optimization problem, which is solved by the alternating direction method of multipliers (ADMM) algorithm. We then unfold the ADMM algorithm into a fixed-length neural network that reduces the iterations dramatically and does not require additional location information of nodes. Experimental results on a realworld dataset demonstrate that the proposed method can achieve faster convergence speed than the baseline ADMM algorithm with slight accuracy loss.

We focus on the convergence rate of the alternating direction method of multipliers (ADMM) in a complex domain. First, the complex form of variational inequality (VI) is established by using the Wirtinger calculus technique. Second, the \(O(1/K)\) convergence rate of the ADMM in a complex domain is provided. Third, the ADMM in a complex domain is applied to the least absolute shrinkage and selectionator operator (LASSO). Finally, numerical simulations are provided to show that ADMM in a complex domain has the \(O(1/K)\) convergence rate and that it has certain advantages compared with the ADMM in a real domain. doi:10.1017/S1446181118000184

For low-density parity-check (LDPC) codes decoding by alternating direction method of multipliers (ADMM), the layered scheduling sequentially updates the messages of check nodes one by one. Though the layered scheduling can speed up the convergence rates, it may limit the throughput when implementing the ADMM decoder with multi-core systems due to its serial style. To circumvent this problem, a group-based layered scheduling is proposed by updating a group of check node messages at one time. Extensive simulation results for the proposed scheme over two typical LDPC codes with the ADMM penalized decoding algorithm are provided.

Alternating Direction Method of Multiplier (ADMM) has been a popular algorithmic framework for separable optimization problems with linear constraints. For numerical ADMM fail to exploit the particular structure of the problem at hand nor the input data information, leveraging task-specific modules (e.g., neural networks and other data-driven architectures) to extend ADMM is a significant but challenging task. This work focuses on designing a flexible algorithmic framework to incorporate various task-specific modules (with no additional constraints) to improve the performance of ADMM in real-world applications. Specifically, we propose Guidance from Optimality (GO), a new customization strategy, to embed task-specific modules into ADMM (GO-ADMM). By introducing an optimality-based criterion to guide the propagation, GO-ADMM establishes an updating scheme agnostic to the choice of additional modules. The existing task-specific methods just plug their task-specific modules into the numerical iterations in a straightforward manner. Even with some restrictive constraints on the plug-in modules, they can only obtain some relatively weaker convergence properties for the resulted ADMM iterations. Fortunately, without any restrictions on the embedded modules, we prove the convergence of GO-ADMM regarding objective values and constraint violations, and derive the worst-case convergence rate measured by iteration complexity. Extensive experiments are conducted to verify the theoretical results and demonstrate the efficiency of GO-ADMM.

The intensities (i.e., number of protons in monitor unit [MU]) of deliverable proton spots need to be either zero or meet a minimum-MU (MMU) threshold, which is a nonconvex problem. Since the dose rate is proportionally associated with the MMU threshold, higher-dose-rate proton radiation therapy (RT) (e.g., efficient intensity modulated proton therapy (IMPT) and ARC proton therapy, and high-dose-rate-induced FLASH effect needs to solve the MMU problem with larger MMU threshold, which however makes the nonconvex problem more difficult to solve. This work will develop a more effective optimization method based on orthogonal matching pursuit (OMP) for solving the MMU problem with large MMU thresholds, compared to state-of-the-art methods, such as alternating direction method of multipliers (ADMM), proximal gradient descent method (PGD), or stochastic coordinate descent method (SCD). The new method consists of two essential components. First, the iterative convex relaxation (ICR) method is used to determine the active sets for dose-volume planning constraints and decouple the MMU constraint from the rest. Second, a modified OMP optimization algorithm is used to handle the MMU constraint: the non-zero spots are greedily selected via OMP to form the solution set to be optimized, and then a convex constrained subproblem is formed and can be conveniently solved to optimize the spot weights restricted to this solution set via OMP. During this iterative process, the new non-zero spots localized via OMP will be adaptively added to or removed from the optimization objective. The new method via OMP is validated in comparison with ADMM, PGD and SCD for high-dose-rate IMPT, ARC, and FLASH problems of large MMU thresholds, and the results suggest that OMP substantially improved the plan quality from PGD, ADMM and SCD in terms of both target dose conformality (e.g., quantified by max target dose and conformity index) and normal tissue sparing (e.g., mean and max dose). For example, in the brain case, the max target dose for IMPT/ARC/FLASH was 368.0%/358.3%/283.4% respectively for PGD, 154.4%/179.8%/150.0% for ADMM, 134.5%/130.4%/123.0% for SCD, while it was <120% in all scenarios for OMP; compared to PGD/ADMM/SCD, OMP improved the conformity index from 0.42/0.52/0.33 to 0.65 for IMPT and 0.46/0.60/0.61 to 0.83 for ARC. A new OMP-based optimization algorithm is developed to solve the MMU problems with large MMU thresholds, and validated using examples of IMPT, ARC, and FLASH with substantially improved plan quality from ADMM, PGD, and SCD.

The Alternating Direction Method of Multipliers(ADMM) is an important method for machine learning. A large number of stochastic versions of ADMM continue to emerge. But almost all the algorithm focused on the Steepest Descent, which cause slow convergence rate. In our work, we propose a new stochastic alternative direction method of multipliers with conjugate gradient(SCG-ADMM), and prove its convergence rate in ergodic sense. Conjugate gradient descent has a better performance in traditional cases. Experiments on LASSO and Logistics Regression show that our proposed algorithm can achieve state-of-the-art performance in real applications. And we use the linear search method to replace the manual setting of the step length.