Communication-efficient distributed cubic Newton with compressed lazy Hessian

The rapid convergence rate, high fidelity learning outcome and low computational cost are key targets in solving the learning problem of the complex physical system. Guided by physical laws of wave propagation, in full waveform inversion (FWI), we learn the subsurface images through optimizing the media velocity model in a large scale non-linear problem. In this paper, we combine randomized subsampling techniques with a second-order optimization algorithm to propose the Sub-Sampled Newton (SSN) method for learning velocity model of FWI. By incorporating the curvature information, SSN preserves comparable convergence rate to Newtons method and significantly reduces the iteration cost by approximating the Hessian matrix through a non-uniform subsampling scheme. The numerical experiments demonstrate that the proposed SSN method has a faster convergence rate, and achieves a more accurate velocity model in terms of mean squared error than commonly used methods.

To address the convergence issues in the natural occupation optimization of reduced density matrix functional theory (RDMFT), we recently proposed the explicit-by-implicit (EBI) idea to handle the ensemble N-representability constraint (Yao et al. J. Phys. Chem. Lett. 2021, 12, 6788). This work continues to focus on these issues that can affect the reliability of the electronic structure description in RDMFT; further explores the combination of EBI, as well as the (augmented) Lagrangian methods (both LM and ALM), with both first- and second-order numerical optimization algorithms; and carefully evaluates their performances in natural occupation optimizations of various systems, including strongly correlated systems and large molecules. By comparing both converged energies and elapsed times, it can be seen that the LM and ALM have serious convergence issues for systems of different sizes. In contrast, the optimizations of EBI can converge to better energies with fewer iterations. However, due to the local convergence nature of the Newton's Method (NM) algorithm, EBI@NM still suffers from the local minimum issue for both strongly correlated systems and large molecules. Overall, the combination of EBI with the simple first-order algorithm of gradient descent (GD), namely EBI@GD, consistently provides the lowest converged energies for different types of systems, with the lowest computational scaling. These tests demonstrate the advantages of EBI in the calculations of transition states, strongly correlated systems, and large molecules. Meanwhile, the insights gained from this work are helpful to further develop more efficient algorithms for RDMFT.

In order to improve the accuracy of consensus filters, in this paper, we propose a second-order consensus filtering algorithm based on the pseudo two-hop distributed consensus algorithm, we analyze the performance of the second-order consensus filtering algorithm, and prove that the second-order consensus filtering algorithm is convergent. We analyze the filtering accuracy of the second-order consensus filtering algorithm, and make a comparison for filtering accuracy between the first-order consensus filtering algorithm and the second-order consensus filtering algorithm, simultaneously, simulation is proposed, the results show the second-order consensus filtering algorithm is convergent, and the filtering accuracy is higher than that of the first-order consensus algorithm.

The dynamic programming formulation of the forward principle of optimality in the solution of optimal control problems results in a partial differential equation with initial boundary condition whose solution is independent of terminal cost and terminal constraints. Based on this property, two computational algorithms are described. The first-order algorithm with minimum computer storage requirements uses only integration of a system of differential equations with specified initial conditions and numerical minimization in finite-dimensional space. The second-order algorithm is based on the differential dynamic programming approach. Either of the two algorithms may be used for problems with nondifferentiable terminal cost or terminal constraints, and the solution of problems with complicated terminal conditions (e.g., with free terminal time) is greatly simplified.

This article investigates a second-order multi-agent distributed optimisation algorithm on matrix-weighted networks. To address the limitations of traditional methods on matrix-weighted networks, a novel second-order distributed optimisation algorithm is proposed. By constructing auxiliary variables, the algorithm eliminates the requirement for velocity information updates. Under the proposed control protocol, algebraic and graph-theoretic conditions for achieving optimisation consensus are established based on matrix theory and Barbalat's lemma. Specifically, the given parameter conditions and local cost functions are convex; the second-order consensus at the global optimal solution is achieved when the Laplacian matrix's null space spans the consensus subspace or the matrix-weighted graph contains a positive spanning tree. Finally, a numerical experiment is conducted to validate the effectiveness of proposed algorithms.

Federated learning (FL) is a promising learning paradigm that can tackle the increasingly prominent isolated data islands problem while keeping users' data locally with privacy and security guarantees. However, FL could result in task-oriented data traffic flows over wireless networks with limited radio resources. To design communication-efficient FL, most of the existing studies employ the first-order federated optimization approach that has a slow convergence rate. This however results in excessive communication rounds for local model updates between the edge devices and edge server. To address this issue, in this paper, we instead propose a novel over-the-air second-order federated optimization algorithm to simultaneously reduce the communication rounds and enable low-latency global model aggregation. This is achieved by exploiting the waveform superposition property of a multi-access channel to implement the distributed second-order optimization algorithm over wireless networks. The convergence behavior of the proposed algorithm is further characterized, which reveals a linear-quadratic convergence rate with an accumulative error term in each iteration. We thus propose a system optimization approach to minimize the accumulated error gap by joint device selection and beamforming design. Numerical results demonstrate the system and communication efficiency compared with the state-of-the-art approaches.

A numerical algorithm for the solution of a class of optimal control problems is presented. This class is characterized as being linear-quadratic in the control, but not necessarily in the state. The algorithm is efficient in that it possesses a high rate of convergence (like Newton's method), but without being sensitive to initialization. In this respect, it combines the advantages of both the first- and second-order algorithms within the framework of a single algorithm. In addition, the algorithm requires less computation and storage than Newton's method.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

A second-order algorithm is presented for the solution of continuous-time nonlinear optimal control problems. The algorithm is an adaptation of the trust region modifications of Newton's method and solves at each iteration a linear-quadratic control problem with an additional constraint. Under some assumptions, the proposed algorithm is shown to possess a global convergence property. A numerical example is presented to illustrate the method.

While there already exist randomized subspace Newton methods that restrict the search direction to a random subspace for a convex function, we propose a randomized subspace regularized Newton method for a non-convex function and more generally we investigate thoroughly, for the first time, the local convergence rate of the randomized subspace Newton method. In our proposed algorithm, we use a modified Hessian of the function restricted to some random subspace so that, with high probability, the function value decreases at each iteration, even when the objective function is non-convex. We show that our method has global convergence under appropriate assumptions and its convergence rate is the same as that of the full regularized Newton method. Furthermore, we obtain a local linear convergence rate under some additional assumptions, and prove that this rate is the best we can hope, in general, when using a random subspace. We furthermore prove that if the Hessian, at the local optimum, is rank deficient then super-linear convergence holds.

Epidemiological models have tremendous potential to forecast disease burden and quantify the impact of interventions. Detailed models are increasingly popular, however these models tend to be stochastic and very costly to evaluate. Fortunately, readily available high-performance cloud computing now means that these models can be evaluated many times in parallel. Here, we briefly describe PSPO, an extension to Spall's second-order stochastic optimization algorithm, Simultaneous Perturbation Stochastic Approximation (SPSA), that takes full advantage of parallel computing environments. The main focus of this work is on the use of PSPO to maximize the pseudo-likelihood of a stochastic epidemiological model to data from a 1861 measles outbreak in Hagelloch, Germany. Results indicate that PSPO far outperforms gradient ascent and SPSA on this challenging likelihood maximization problem.

Distributed learning (DL) is considered a cornerstone of intelligence enabler, since it allows for collaborative training without the necessity for local clients to share raw data with other parties, thereby preserving privacy and security. Integrating DL into the 6G networks requires coexistence design with existing services such as high-bandwidth (HB) traffic like eMBB. Current designs in the literature mainly focus on communication round (CR)-wise designs that assume a fixed resource allocation during each CR. However, fixed resource allocation within a CR is a highly inefficient and inaccurate representation of the system's realistic behavior. This is due to the heterogeneous nature of the system, where clients inherently need to access the network at different times. This work zooms into one arbitrary communication round and demonstrates the importance of considering a time-dependent resource-sharing design with HB traffic. We propose a time-dependent optimization problem for minimizing the consumed time and energy by DL within the CR. Due to its intractability, a session-based optimization problem has been proposed assuming a large-scale coherence time. An iterative algorithm has been designed to solve such problems and simulation results confirm the importance of such efficient and accurate integration design.

A hybrid algorithm consisting of a Gauss--Newton method and a second-order method for solving constrained and weighted nonlinear least squares problems is developed, analyzed, and tested. One of the advantages of the algorithm is that arbitrarily large weights can be handled and that the weights in the merit function do not get unnecessarily large when the iterates diverge from a saddle point. The local convergence properties for the Gauss--Newton method are thoroughly analyzed and simple ways of estimating and calculating the local convergence rate for the Gauss--Newton method are given. Under the assumption that the constrained and weighted linear least squares subproblems attained in the Gauss--Newton method are not too ill conditioned, global convergence towards a first-order KKT point is proved.

It is well-known that primal first-order algorithms achieve sublinear (linear) convergence for smooth convex (smooth strongly convex) constrained minimization. However, these methods encounter numerical difficulties when the primal feasible set is complicated, since they require exact projection onto this set. Algorithmic alternatives to convex problems with complicated feasible set are the dual first-order methods. Dual methods are able to handle easily complicated constraints, but they have difficulties in convergence when the norm of the optimal Lagrange multiplier is large, since this norm appears linearly in the convergence estimates of these methods. Moreover, they have typically sublinear convergence rate in an average primal sequence, even when the primal problem has smooth and strongly convex objective function. Motivated by these issues, in this paper, we analyze the convergence of primal first-order methods with inexact projections for solving constrained convex problems with smooth and then strongly convex objective function. In particular, we consider the inexact variants of Projected Gradient and Projected Fast Gradient methods, where instead of computing an exact projection onto the complicated primal feasible set, an approximate projection, not necessarily feasible, is used. We prove that we can still achieve similar convergence rates for these inexact projection first-order algorithms with those given in the exact projection settings, provided that the approximate projection is sufficiently accurate. Our convergence analysis allows us to derive explicitly the accuracy of the inexact projection and the number of iterations we need to perform in order to obtain an approximate solution for our convex problem. Finally, practical performance on random quadratic problems show encouraging results.

In order to improve the accuracy and the convergence rate of distributed consensus under quantized communication, in the paper, based on adaptive quantization scheme, we propose the fast second-order distributed consensus algorithm. We analyze the convergence performance. The fast second-order distributed consensus with adaptive quantization achieves a consensus in a mean square sense, and the consensus is equal to the average of the initial states. Simultaneously, Simulations are done about the fast second-order distributed consensus based on adaptive quantization. Results show that the fast second-order distributed consensus algorithm based on adaptive quantization can reach an average consensus, and its convergence rate is higher than that of the first-order adaptive quantized distributed consensus algorithm, moreover, the mean square errors are smaller within the finite steps.

This paper focuses on the large-scale composite optimization problem, which is the sum of a finite number of smooth (possibly nonconvex) functions and a convex (possibly nonsmooth) function. A common method to solve this problem is proximal incremental aggregated gradient (PIAG) method (N. D. Vanli, M. Gurbuzbalaban and A. Ozdaglar, SIAM Journal on Optimization, 2018, 28(2): 1282–1300), which exploits the gradient information from previous iterations to approximate the gradient at the current point. However, as many first-order algorithms, it may suffer from slow convergence. To address this issue, this paper proposes an inexact variable metric variant of incremental aggregated Forward-Backward (iVMFB-IAG) method. This methods incorporates a variable metric strategy to accelerate the PIAG method and introduces an inexact criterion to enhance its practicality. Under the Kurdyka-Łojasiewicz (KL) property and the uniformly bounded positive definiteness of the scaling matrix, the sequence generated by the iVMFB-IAG method converges globally to the critical point of the problem. Additionally, the convergence rate is established under the KL framework. Moreover, under certain restrictive conditions, the method exhibits local superlinear convergence. Through the numerical experiments on image reconstruction applications, we demonstrate the effectiveness of the iVMFB-IAG method by restoring two distinct images.