Explicit-by-Implicit Treatment of Natural Orbital Occupations Using First- and Second-Order Optimization Algorithms: A Comparative Study.

The paper proposes and develops new globally convergent algorithms of the generalized damped Newton type for solving important classes of nonsmooth optimization problems. These algorithms are based on the theory and calculations of second-order subdifferentials of nonsmooth functions with employing the machinery of second-order variational analysis and generalized differentiation. First we develop a globally superlinearly convergent damped Newton-type algorithm for the class of continuously differentiable functions with Lipschitzian gradients, which are nonsmooth of second order. Then we design such a globally convergent algorithm to solve a structured class of nonsmooth quadratic composite problems with extended-real-valued cost functions, which typically arise in machine learning and statistics. Finally, we present the results of numerical experiments and compare the performance of our main algorithm applied to an important class of Lasso problems with those achieved by other first-order and second-order optimization algorithms.

Communication-efficient distributed cubic Newton with compressed lazy Hessian

Feedforward neural networks (FNN's) have been proposed to solve complex problems in pattern recognition and classification and function approximation. Despite the general success of learning methods for FNN's, such as the backpropagation (BP) algorithm, second-order optimization algorithms and layer-wise learning algorithms, several drawbacks remain to be overcome. In particular, two major drawbacks are convergence to a local minima and long learning time. In this paper we propose an efficient learning method for a FNN that combines the BP strategy and optimization layer by layer. More precisely, we construct the layer-wise optimization method using the Taylor series expansion of nonlinear operators describing a FNN and propose to update weights of each layer by the BP-based Kaczmarz iterative procedure. The experimental results show that the new learning algorithm is stable, it reduces the learning time and demonstrates improvement of generalization results in comparison with other well-known methods.

This paper presents a new method that addresses measurement origin uncertainty. Measurement origin uncertainty occurs when the object a measurement originated from is not clear. The systems considered contain multiple bodies which are dynamically indistinguishable other than initial conditions. Each measurement originates from one of the bodies in the system. In the past, recursive data association methods have been used to address problems of this nature. A new technique is presented which treats the measurement association problem as a batch post-processing problem. Reformulating the problem as such, it is possible to transform the data association problem into a trajectory optimization problem. From this point of view it is then possible to solve the measurement association problem using first- and second-order optimization algorithms that rely on having first- and second-order derivatives for cost functions that depend on impulsive trajectories.

Functional to anatomical brain image registration is needed for accurate localization of brain activation maps. Due to the presence of nonlinear distortions, it is more effective to consider non-rigid transformations to achieve such a registration. In this paper, a non-rigid registration technique based on the B-spline free-form deformation model and mutual information similarity measure is introduced. An optimization formulation is devised to achieve a fast and robust registration. This formulation differs from the previous formulations by utilizing a limited-memory, second-order optimization algorithm instead of the usual first-order gradient-based algorithms. It also enforces hard parameter constraints instead of constraints based upon physics or Jacobian smoothness. The results obtained indicate that this registration technique provides improvements over rigid and affine techniques when registering functional to anatomical magnetic resonance brain images.

We present an approach for efficient numerical solution of optimization problems governed by parabolic partial differential equations. The main ingredients are: space-time finite element discretization, second-order optimization algorithms, and storage reduction techniques. We discuss the combination of these components for the solution of large-scale optimization problems.

Federated learning (FL) is a promising learning paradigm that can tackle the increasingly prominent isolated data islands problem while keeping users' data locally with privacy and security guarantees. However, FL could result in task-oriented data traffic flows over wireless networks with limited radio resources. To design communication-efficient FL, most of the existing studies employ the first-order federated optimization approach that has a slow convergence rate. This however results in excessive communication rounds for local model updates between the edge devices and edge server. To address this issue, in this paper, we instead propose a novel over-the-air second-order federated optimization algorithm to simultaneously reduce the communication rounds and enable low-latency global model aggregation. This is achieved by exploiting the waveform superposition property of a multi-access channel to implement the distributed second-order optimization algorithm over wireless networks. The convergence behavior of the proposed algorithm is further characterized, which reveals a linear-quadratic convergence rate with an accumulative error term in each iteration. We thus propose a system optimization approach to minimize the accumulated error gap by joint device selection and beamforming design. Numerical results demonstrate the system and communication efficiency compared with the state-of-the-art approaches.

The rapid convergence rate, high fidelity learning outcome and low computational cost are key targets in solving the learning problem of the complex physical system. Guided by physical laws of wave propagation, in full waveform inversion (FWI), we learn the subsurface images through optimizing the media velocity model in a large scale non-linear problem. In this paper, we combine randomized subsampling techniques with a second-order optimization algorithm to propose the Sub-Sampled Newton (SSN) method for learning velocity model of FWI. By incorporating the curvature information, SSN preserves comparable convergence rate to Newtons method and significantly reduces the iteration cost by approximating the Hessian matrix through a non-uniform subsampling scheme. The numerical experiments demonstrate that the proposed SSN method has a faster convergence rate, and achieves a more accurate velocity model in terms of mean squared error than commonly used methods.

We design a non-convex second-order optimization algorithm that is guaranteed to return an approximate local minimum in time which scales linearly in the underlying dimension and the number of training examples. The time complexity of our algorithm to find an approximate local minimum is even faster than that of gradient descent to find a critical point. Our algorithm applies to a general class of optimization problems including training a neural network and other non-convex objectives arising in machine learning.

This article investigates a second-order multi-agent distributed optimisation algorithm on matrix-weighted networks. To address the limitations of traditional methods on matrix-weighted networks, a novel second-order distributed optimisation algorithm is proposed. By constructing auxiliary variables, the algorithm eliminates the requirement for velocity information updates. Under the proposed control protocol, algebraic and graph-theoretic conditions for achieving optimisation consensus are established based on matrix theory and Barbalat's lemma. Specifically, the given parameter conditions and local cost functions are convex; the second-order consensus at the global optimal solution is achieved when the Laplacian matrix's null space spans the consensus subspace or the matrix-weighted graph contains a positive spanning tree. Finally, a numerical experiment is conducted to validate the effectiveness of proposed algorithms.

Epidemiological models have tremendous potential to forecast disease burden and quantify the impact of interventions. Detailed models are increasingly popular, however these models tend to be stochastic and very costly to evaluate. Fortunately, readily available high-performance cloud computing now means that these models can be evaluated many times in parallel. Here, we briefly describe PSPO, an extension to Spall's second-order stochastic optimization algorithm, Simultaneous Perturbation Stochastic Approximation (SPSA), that takes full advantage of parallel computing environments. The main focus of this work is on the use of PSPO to maximize the pseudo-likelihood of a stochastic epidemiological model to data from a 1861 measles outbreak in Hagelloch, Germany. Results indicate that PSPO far outperforms gradient ascent and SPSA on this challenging likelihood maximization problem.

Convergence acceleration is always a critical issue in distributed multi-area scheduling. The critical region projection (CRP) algorithm based on multi-parametric quadratic programming (MPQP) shows better convergence performance for multi-area static economic dispatch. However, its imperfect decomposition framework cannot be directly applied to dynamic economic dispatch. Therefore, this paper modifies the CRP algorithm to achieve fast distributed multi-area intra-day economic dispatch (MAIDED). First, we introduce slack variables to each area and add the corresponding penalty term into their objective functions to present a primal decomposition framework with penalty relaxation. It is more realistic than the decomposition framework of traditional CRP (TCRP). Then, an iterative algorithm of double spatial scale search (DSSS) is developed to improve the convergence rate of distributed solving based on the similar optimal value function in adjacent critical regions (CRs). Moreover, we design an initial value selection method based on data fitting to further reduce the number of iterations. Finally, three interconnected power systems of different sizes are used for numerical testing to demonstrate that the proposed modified CRP (MCRP) algorithm can meet the practical application and has higher convergence efficiency. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the problem of distributed MAIDED for power systems but it also applies to other multi-agent networks with a coordinator. The convergence speed of the existing distributed optimization algorithm is slow, which increases the risk of communication failure and attack. Meanwhile, more iterations will result in increased communication and computing costs. This paper modified the TCRP algorithm for high-efficient distributed MAIDED. The penalty relaxation is used to ensure that the decomposition framework is consistent with the actual system operation. The DSSS algorithm and the initial value selection method based on data fitting are designed to effectively reduce the number of iterations. In this paper, the DSSS is that the coordinator enlarges the CRs uploaded from each area to perform a rough optimization for approaching the global optimal solution fast, and then in the next iteration, conducts a precise optimization like the TCRP algorithm to determine the precise optimal solution. The numerical testing of the interconnected power systems of different sizes indicates that the MCRP algorithm has fewer iterations and calculation times than the TCRP algorithm. Future work will extend the MCRP algorithm to solve the distributed multi-area unit commitment problem.

Optimization is an important branch which aims to conceptualize, analyze, and solve problems of minimization or maximization of a function on a specific dataset. Several optimization algorithms are discussed in machine learning and particularly in deep learning (DL) based systems such as the Gradient Descent (GD) algorithm. Given the importance and the efficiency of the gradient descent algorithm, several research works have made it possible to optimize it and demonstrate its performance, Otherwise, regularity of a train is essential to ensure the continuity of the entire rail system. Non-regularity can spread quickly and influence the rest of the means of transport: rail, road, air, navy etc. In this paper, we perform a comparative study of different optimizations algorithms which are largely used in context of machine learning on the prediction of the regularity of trains, the data used is publicly available. The optimization algorithms studied are Momentum, Adagrad, RMSprop Adam and Adamax. In our context, the overall experimental results obtained show that RMSprop performed better compared to other optimization techniques, while Momentum represents the lowest performances to improve regularity.KeywordsMachine learningOptimizationGradient descentRegularityTrain

A numerical algorithm for the solution of a class of optimal control problems is presented. This class is characterized as being linear-quadratic in the control, but not necessarily in the state. The algorithm is efficient in that it possesses a high rate of convergence (like Newton's method), but without being sensitive to initialization. In this respect, it combines the advantages of both the first- and second-order algorithms within the framework of a single algorithm. In addition, the algorithm requires less computation and storage than Newton's method.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This research tries to stabilise a three degree-of-freedom series-type double inverted pendulum system by employing an adaptive state feedback control approach optimised by a particle swarm optimisation (PSO) algorithm. To this end, the dynamical equations are derived via Lagrangian method and linearised by utilising approximation approaches. Then, the state feedback scheme is implemented to control the system states, that is, the angle and angular velocity of the first and second poles as well as the position and velocity of the cart. In order to timely regulate the control gains, the adaptation rules based on the gradient descent method and sliding surface relations are applied. The particle swarm optimisation algorithm is implemented to optimally tune the design parameters of the controller based on a proper objective function defined as summation of integrals of absolute values of the system errors. The time responses of the angles of the poles and the position of the cart clearly depict the feasibility and efficiency of the introduced control strategy to stabilise the system from different initial conditions to the final desired values.