Graph-aware Weighted Hybrid ADMM for Fast Decentralized Optimization

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

Distributed machine learning has gained lots of attention due to the rapid growth of data. In this paper, we focus regularized empirical risk minimization problems, and propose two novel Distributed Accelerated Alternating Direction Method of Multipliers (D-A2DM2) algorithms for distributed classification. Based on the framework of Alternating Direction Method of Multipliers (ADMM), we decentralize the distributed classification problem as a global consensus optimization problem with a series of sub-problems. In D-A2DM2, we exploit ADMM with variance reduction for sub-problem optimization in parallel. Taking global update and local update into consideration respectively, we propose two acceleration mechanisms in the framework of D-A2DM2. In particular, inspired by Nesterov's accelerated gradient descent, we utilize it for global update to further improve time efficiency. Moreover, we also introduce Nesterov's acceleration for local update, and develop the corrected local update and symmetric dual update to accelerate the convergence with only a little change in the computational effort. Theoretically, D-A2DM2 has a linear convergence rate. Empirically, experimental results show that D-A2DM2 converge faster than existing distributed ADMM-based classification, and could be a highly efficient algorithm for practical use.

Recently, the alternating direction method of multipliers (ADMM) has been used effectively to solve the multi-agent unconstrained optimization problems, where the objective function is the sum of privately known local objective functions of agents. In this paper, first, with the help of the edge-node incidence matrix, an unconstrained optimization problem is transformed into an equivalent optimization problem with only equality constraint and, thus, can be dealt with the ADMM conveniently. Second, a novel distributed inexact consensus ADMM is proposed to enable the agents to reach consensus on the optimal solution of the optimization problem. At the same time, the analysis of the linear convergence of the proposed algorithm is also provided under some mild conditions. Finally, some simulation results are presented to demonstrate the better effectiveness of the proposed algorithm than the standard consensus-based ADMM algorithm.

Transform Invariant Low-Rank Textures, referred to as TILT, can accurately and robustly extract textural or geometric information in a 3D from user-specified windows in 2D in spite of significant corruptions and warping. It was discovered that the task can be characterized, both theoretically and numerically, by solving a sequence of matrix nuclear-norm and $\ell_1$-norm involved convex minimization problems. For solving this problem, the direct extension of Alternating Direction Method of Multipliers (ADMM) in an usual Gauss-Seidel manner often performs numerically well in practice but there is no theoretical guarantee on its convergence. In this paper, we resolve this dilemma by using the novel symmetric Gauss-Seidel (sGS) based ADMM developed by Li, Sun \& Toh (Math. Prog. 2016). The sGS-ADMM is guaranteed to converge and we shall demonstrate in this paper that it is also practically efficient than the directly extended ADMM. When the sGS technique is applied to this particular problem, we show that only one variable needs to be re-updated, and this updating hardly imposes any excessive computational cost. The sGS decomposition theorem of Li, Sun \& Toh (arXiv: 1703.06629) establishes the equivalent between sGS-ADMM and the classical ADMM with an additional semi-proximal term, so the convergence result is followed directly. Extensive experiments illustrate that the sGS-ADMM and its generalized variant have superior numerical efficiency over the directly extended ADMM.

Multi-block linear constrained separable convex minimizations are ubiquitous and have been drawing increasing attention in recent researches. The alternating direction method of multipliers (ADMM) has been well studied and used in the literature for the two-block case. However, the direct extension of the ADMM to the multi-block case is not necessarily convergent. ADMM with Gaussian Back Substitution and ADMM with Prox-Parallel Splitting are two useful schemes to deal with the multi-block situation. Nevertheless, only a sublinear convergence rate was given in previous studies. In this paper, we prove the linear convergence rate of these two schemes under some assumptions. The proofs mainly depend on the variational inequalities.

This paper develops efficient algorithms for distributed average consensus with quantized communication using the alternating direction method of multipliers (ADMM). When rounding quantization is employed, a distributed ADMM algorithm is shown to converge to a consensus within 3 + ⌈log1+δ Ω⌉ iterations where δ > 0 depends on the network topology and O is a polynomial of the quantization resolution, the agents' data and the network topology. A tight upper bound on the consensus error is also obtained, which depends only on the quantization resolution and the average degree of the graph. This bound is much preferred in large scale networks over existing algorithms whose consensus errors are increasing in the range of agents' data, the quantization resolution, and the number of agents. To minimize the consensus error, our final algorithm uses dithered quantization to obtain a good starting point and then adopts rounding quantization to reach a consensus. Simulations show that the consensus error of this algorithm is typically less than one quantization resolution for all connected networks with agents' data of arbitrary magnitudes.

We discuss an online decentralized decision making problem where the agents are coupled with affine inequality constraints. Alternating Direction Method of Multipliers (ADMM) is used as the computation engine and we discuss the convergence of the algorithm in an online setting. To be specific, when decisions have to be made sequentially with a fixed time step, there might not be enough time for the ADMM to converge before the scenario changes and the decision needs to be updated. In this case, a suboptimal solution is employed and we analyze the optimality gap given the convergence condition. Moreover, in many cases, the decision making problem changes gradually over time. We propose a warm-start scheme to accelerate the convergence of ADMM and analyze the benefit of the warm-start. The proposed method is demonstrated in a decentralized multiagent control barrier function problem with simulation.

In this work, we leverage the alternating direction method of multipliers (ADMM) framework to solve task assignment for a multi-robot team. While ADMM is a well-established method, it has yet to be utilized in multi-robot problems as the standard formulation requires a centralized update step, a paradigm that conflicts with decentralization as a means of robustness. Therefore, we describe the formulation of separable optimizations in order to produce decentralized ADMM algorithms. Here, our aim is to provide an additional tool for solving cooperative team-based problems in robotics. For the decentralized algorithms, we discuss the conditions for convergence to the optimal centralized solution. We present simulation results for task assignment to demonstrate the benefits of ADMM compared to state-of-the-art methods.KeywordsADMMTask assignmentConsensus methodsCooperative teams

Parameter estimation of damped exponential signals has wide applications including fault detection and system parameter identification, etc. However, existing methods for estimating parameters of damped exponentials are either sensitive to noise or restricted to dealing with a certain type of noise such as Gaussian noise. In this paper we aim to estimate parameters of damped exponentials from contaminated signal, i.e., a mixture of damped exponentials, random Gaussian noise, and spike interference. We propose two robust approaches, a convex one solved by the alternating direction method of multipliers (ADMM) and a non-convex one solved by coordinate descent, to recovering a low-rank Hankel matrix of damped exponentials from noisy measurements for further parameter estimation using the matrix pencil technique. Numerical experiments show that our proposed methods outperform classical ones in detecting small damped fault signatures from noisy measurements. While the convex approach is amenable to theoretical analysis and global convergence guarantees, the non-convex one exhibits more robustness and computational efficiency.

Integrated sensing and communication (ISAC) has been envisioned to play a more important role in future wireless networks. However, the design of ISAC networks is challenging, especially when there are multiple communication and sensing (C&S) nodes and multiple sensing targets. We investigate a multi-base station (BS) ISAC network in which multiple BSs equipped with multiple antennas simultaneously provide C&S services for multiple ground communication users (CUs) and targets. To enhance the overall performance of C&S, we formulate a joint user association (UA) and multi-BS transmit beamforming optimization problem with the objective of maximizing the total sum rate of all CUs while ensuring both the minimum target detection and parameter estimation requirements in terms of the radar signal-to-noise ratio (SNR) and the Cramér-Rao bound (CRB), respectively. To efficiently solve the highly non-convex mixed integer nonlinear programming (MINLP) optimization problem, we propose an alternating optimization (AO)-based algorithm that decomposes the problem into two sub-problems, i.e., UA optimization and multi-BS transmit beamforming optimization. Inspired by the huge potential of large language models (LLMs) for prediction and inference, we propose a unified framework integrating LLMs with convex-based optimization methods to benefit from the theoretical rigor and convergence guarantees of convex-based methods, and the adaptability and flexibility of LLMs. First, we propose a comprehensive design of prompt engineering based on in-context, few-shot, chain of thought, and self-reflection techniques to guide LLMs in solving the binary integer programming UA optimization problem. Second, we utilize convex-based optimization methods to handle the non-convex beamforming optimization problem based on fractional programming (FP), majorization minimization (MM), and the alternating direction method of multipliers (ADMM) with an optimized UA from LLMs. Numerical results demonstrate that our proposed LLM-enabled AO-based algorithm achieves fast convergence and near upper-bound performance with the GPT-o1 model, outperforming various benchmark schemes, which shows the advantages of integrating LLMs into convex-based optimization for wireless networks.

Recently, several convergence rate results for Douglas-Rachford splitting and the alternating direction method of multipliers (ADMM) have been presented in the literature. In this paper, we show global linear convergence rate bounds for Douglas-Rachford splitting and ADMM under strong convexity and smoothness assumptions. We further show that the rate bounds are tight for the class of problems under consideration for all feasible algorithm parameters. For problems that satisfy the assumptions, we show how to select step-size and metric for the algorithm that optimize the derived convergence rate bounds. For problems with a similar structure that do not satisfy the assumptions, we present heuristic step-size and metric selection methods.

Multi-agent systems have gained prominence in various applications for their scalability, flexibility, and efficiency. However, generating coordinated formation trajectories in a distributed manner remains challenging due to the lack of global information or designated leaders. To address this issue, we propose a fully distributed framework based on dynamic game theory. The problem is formulated as a dynamic trajectory game with a potential structure, jointly optimizing collision avoidance, control effort, formation coordination, and goal-reaching objectives. To compute the corresponding Nash equilibrium (NE), we introduce a distributed algorithm termed ADRD. By leveraging the consensus Alternating Direction Method of Multipliers (ADMM), our approach decouples inter-agent interactions. This enables each agent to independently compute its trajectory with theoretical convergence guarantees. Extensive simulations in obstacle-rich environments demonstrate that ADRD produces safe and coordinated trajectories with superior robustness compared to centralized and distributed baselines. Quantitative comparisons show consistent improvements over the proximal iterative best response scheme in formation accuracy, control cost, and overall cost metrics. Scalability experiments with up to 12 agents further validate the effectiveness of the method in larger teams.

This paper proposes a novel distributed approach to solve a new dynamic economic dispatch problem (DEDP) in which environmental cost function and ramp rate constraints are taken into consideration in islanded microgrid. In our proposed optimization model, the environmental cost function with E-exponential term and ramp rate constraints are considered to make the optimization problem more practical. Then a novel fully distributed algorithm is proposed to address the DEDP based on the alternating direction method of multipliers (ADMM) and distributed consensus theory of the multiagents system. A Lambert W function is employed to tackle the E-exponential term in the environmental cost function, which is different from most existing papers which discuss the DEDP only with quadratic cost functions. A parallel projection method on account of ADMM is used to deal with the ramp rate constraints in this paper. In addition, the power balance can be guaranteed every time when the sum of initial powder output is equal to the total demand. Therefore, the proposed algorithm can deal with the supply-demand constraints, capacity limit constraints, and ramp rate constraints. Finally, simulations on IEEE14-bus are introduced to further illustrate the effectiveness of the proposed algorithm.

Decentralized optimal scheduling for integrated community energy system via consensus-based alternating direction method of multipliers

We consider a dynamic time division duplexing wireless network and propose a distributed coordinated beamforming algorithm based on Alternating Direction Method of Multipliers (ADMM) technique assuming the availability of perfect channel state information. Our design objective is to minimize the sum transmit power at the base stations subject to minimum signal-to-interference-plus-noise ratio (SINR) constraints for downlink mobile stations and a maximum interference power threshold for uplink mobile stations. First, we propose a centralized algorithm based on the relaxed Semidefinite Programming (SDP) technique. To obtain the beamforming solution in a distributed way, we further propose a distributed coordinated beamforming algorithm using the ADMM technique. Detailed simulation results are presented to examine the effectiveness of the proposed algorithms. It is shown that the proposed algorithm achieves better performance in terms of the design objective and converges faster than the reference algorithm based on primal decomposition.