A constrained optimization perspective on actor–critic algorithms and application to network routing

Solution of nonlinear optimal control problems on analog parallel networks is proposed. Recurrent neural networks whose dynamic equations have a Lyapunov function are developed. Such circuits relax to an equilibrium which is the minimum of the Lyapunov function. Nonlinear optimal control problems are formulated in terms of a Lyapunov function and thus are solved using the recurrent networks. Convergence for linear and nonlinear classes of problems is considered. The method is demonstrated by developing and simulating a network to solve a nonlinear vibration problem. Simulation results demonstrate solution times are accurate and extremely fast. Solution times are shown to be independent of the size of the problem.

This paper introduces a novel approach for solving high-dimensional nonlinear optimization problems by integrating neural networks into the optimization process. The method leverages the capabilities of neural networks to efficiently handle complex, high-dimensional data and to approximate discrete numerical solutions of nonlinear optimization problems using continuous functions. By combining the nonlinear mapping ability of neural networks with iterative optimization algorithms, the proposed approach provides a superior method to solve nonlinear optimization problems. The paper demonstrates the adaptability of neural network-based solution methods in solving various nonlinear optimization problems and illustrates that the method can be applied to many different problem scenarios. The effectiveness and correctness of the proposed method are demonstrated through examples.

This paper studies the problem of resource management and placement for high performance clouds. It is concerned with the three most important performance metrics: response time, throughput, and utilization as Quality of Service (QoS) metrics defined in a Service Level Agreement (SLA). We propose SLA-based approaches for resource management in clouds. Specifically, we first quantify the metrics of trustworthiness, a percentile of response time, and availability. Then, this paper provides the formulation of cloud resource management as a nonlinear optimization problem subject to SLA requirements and further gives the solution of the problem. Finally, we give a solution of this nonlinear optimization problem and demonstrate the effectiveness of proposed solutions through illustrative examples.

Aim. Development of method for control by orbital spacecraft magnetic cleanliness based on multiple magnetic dipole models using compensation of the initial magnetic field with consideration of magnetic characteristics uncertainty. Methodology. Orbital spacecraft multiple magnetic dipole models calculated as solution of nonlinear minimax optimization problem based on near field measurements for prediction orbital spacecraft far magnetic field magnitude. Nonlinear objective function calculated as the weighted sum of squared residuals between the measured and predicted magnetic field. Weight matrix calculated as inverse covariance matrix of random errors vector. Values of magnetic moments and coordinates of placement of compensating magnetic dipoles for compensation of the orbital spacecraft initial magnetic field also calculated as solution of nonlinear minimax optimization problem. Both solutions of this nonlinear minimax optimization problems calculated based on particle swarm nonlinear optimization algorithms. Results. Results of prediction spacecraft far magnetic field magnitude based on orbital spacecraft multiple magnetic dipole models using near field measurements and compensation of the initial magnetic field with consideration of orbital spacecraft magnetic characteristics uncertainty for ensuring the orbital spacecraft magnetic cleanliness. Originality. The method for control by orbital spacecraft magnetic cleanliness based on multiple magnetic dipole models using compensation of the initial magnetic field with consideration of magnetic characteristics uncertainty is developed. Practical value. An important practical problem of ensuring orbital spacecraft magnetic cleanliness based on orbital spacecraft multiple magnetic dipole models using near field measurements and compensation of the initial magnetic field with consideration of orbital spacecraft magnetic characteristics uncertainty solved.

Aim. Development of method for prediction and control the microsatellite magnetic cleanliness taking into account the uncertainties of the magnetic characteristics of the microsatellite, based on calculation the magnetic field spatial spherical harmonics in the area of the onboard magnetometer installation and using compensating multipoles. Methodology. Spatial spherical harmonics of microsatellite magnetic field in the area of the onboard magnetometer installation calculated as solution of nonlinear minimax optimization problem based on near field measurements for prediction far spacecraft magnetic field magnitude. Nonlinear objective function calculated as the weighted sum of squared residuals between the measured and predicted magnetic field. Values of the compensating dipoles, quadrupoles and octupoles and coordinates of them placement inside the spaceship for compensation of the dipoles, quadrupoles and octupoles components of the microsatellite initial magnetic field also calculated as solution of nonlinear minimax optimization problem. Both solutions of this nonlinear minimax optimization problems calculated based on particle swarm nonlinear optimization algorithms. Results. Results of prediction spacecraft far magnetic field magnitude based on spacecraft spatial spherical harmonics of the magnetic field using near field measurements and compensation of the dipoles, quadrupoles and octupoles components of the initial magnetic field with consideration of spacecraft magnetic characteristics uncertainty for ensuring the microsatellite magnetic cleanliness. Originality. The method for prediction and control by spacecraft magnetic cleanliness based on calculation spatial spherical harmonics of the magnetic field in the area of the onboard magnetometer installation using compensation of the dipoles, quadrupoles and octupoles components of the initial magnetic field with consideration of magnetic characteristics uncertainty is developed. Practical value. The important practical problem of ensuring the magnetic cleanliness of the «Sich-2» microsatellite family based on the spatial spherical harmonics of the magnetic field model using the compensation of the dipole, quadrupole and octupole components of the output magnetic field of the sensor for the kinetic parameters of the neutral component of the space plasma at the point of installation of the on-board magnetometer LEMI-016 by setting the compensating dipole, quadrupole and octupole with consideration of spacecraft magnetic characteristics uncertainty solved.

A large class of sequential decision-making problems under uncertainty, with broad applications from preventive maintenance to event-triggered control can be modeled in the framework of semi-Markov decision processes (SMDPs). Unlike Markov decision processes (MDPs), SMDPs are underexplored in the online and reinforcement learning (RL) settings. In this paper, we extend the well-known deterministic policy gradient (DPG) theorem in MDPs to SMDPs under average-reward criterion. The existing stochastic policy gradient methods not only require, in general, a large number of samples for training, but they also suffer from high variance in the gradient estimation when applied to problems with deterministic optimal policy. Our DPG method can potentially remedy these issues. On the basis of this method and depending on the choice of a critic, different actor–critic algorithms can easily be developed in the RL setup. We present two example actor–critic algorithms. Both algorithms employ our developed policy gradient theorem for their actors, but use two different critics; one uses a simple SARSA update while the other one uses the same on-policy update but with compatible function approximators. We demonstrate the efficacy of our method both mathematically and via simulations.

B ackground. Non-linear optimization problems for the operation of large space or functionally distributed systems with independently functioning subsystems and restrictions on some of the common resources are considered. O bjective. The aim is to build an effective approach to solving nonlinear optimization problems of block structure with coupling constraints on the basis of the combination of approximating nonlinear optimization methods and decomposition techniques. M ethods. The three-step iterative scheme has been proposed. On the upper level the original problem is replaced by a sequence of approximating problems with additively-separable objective function and linear constraints. On the second level, coordinating problems, which are formed by the information received at the lowest level from local block problems, are solved. R esults. The algorithm, which is a combination of the linearization method of B.Pshenichniy and the dual decomposition, has been built. The structure of the dual coordination problem has been described. Estimation of the convergence rate has been carried out. Сonclusions. The proposed approach and the algorithm constructed based on it can be applied to a wide range of problems associated with optimal allocation of limited resources in the large block-structured systems.

This paper deals with the solution of practical nonlinear optimization problems which arise especially in the fields of engineering design and production planning. In their mathematical description these problems frequently consist of highly nonlinear objective functions and constraints whose evaluation may lead to time-consuming simulation runs. In these cases assumptions about unimodality, convexity, and smoothness of well-known solution methods in nonlinear optimization are mostly invalid. An intuitive approach to overcoming these problems would be to apply a simultaneous combination of different optimization algorithms. In this situation the basis for a more reliable and even faster solution is the controlled information exchange on the continuing solution progress between the participating and parallel running methods. This idea has been implemented on a multiprocessor system with distributed memory for the solution of nonlinear optimization problems. Based on previous experiences with different algorithms, a coarse-grained parallelization approach under asynchronous control has been developed.

We consider the problem of optimal allocation of measurement resources, when: (1) the total measurement budget and time duration of measurements are fixed, and (2) the cost of an individual measurement varies inversely with the (controllable) measurement accuracy. The objective is to determine the time-distribution of measurement variances that minimizes a measure of error in estimating a discrete-time, vector stochastic process with known auto-correlation matrix using a linear estimator. The metric of estimation error is the trace of weighted sum of estimation error covariance matrices at various time indices. We show that this problem reduces to a nonlinear optimization problem with linear equality and inequality constraints. The solution to this problem is obtained via a variation of the projected Newton method. For the special case when the vector stochastic process is the state of a linear, finite-dimensional stochastic system, the problem reduces to the solution of a nonlinear optimal control problem. In this case, the gradient and Hessian with respect to the measurement costs are obtained via the solution of a two-point boundary value problem and the resulting optimization problem is solved via a variation of the projected Newton method. The proposed method is illustrated using four examples. >

Adaptive nearly optimal control for a class of continuous-time nonaffine nonlinear systems with inequality constraints

We present a collection of computer programs for the solution of linear and nonlinear semidefinite optimization problems. After briefly discussing the underlying algorithm, the generalized augmented Lagrangian method, we describe details of the specific programs for linear, bilinear and general nonlinear semidefinite optimization problems. For each of the programs we present typical application areas and examples.KeywordsPennonAugmented LagrangianCholesky MethodHessian-vector ProductsLinear Semidefinite Programming Problem (LSDP)These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, a Hopfiled neural network for nonlinear constrained optimization problem is discussed. The energy function for the nonlinear neural network with its neural dynamics is defined based on penalty function with two-order continuous differential. The system of the neural network is stable, and its equilibrium point of the neural dynamics is also an approximately solution for nonlinear constrained optimization problem. Based on the relationship between the equilibrium points and the energy function, an algorithm is developed for computing an equilibrium point of the system or an optimal solution to its optimization problem. The efficiency of the algorithm is illustrated with the numerical examples.

A second-order algorithm is presented for the solution of continuous-time nonlinear optimal control problems. The algorithm is an adaptation of the trust region modifications of Newton's method and solves at each iteration a linear-quadratic control problem with an additional constraint. Under some assumptions, the proposed algorithm is shown to possess a global convergence property. A numerical example is presented to illustrate the method.

This article considers the decentralized stabilization problem of continuous-time nonlinear systems subject to unmatched interconnections and asymmetric input constraints. Initially, with the nonquadratic value functions being introduced to constrained auxiliary subsystems, the decentralized stabilization problem is converted into an array of nonlinear optimal control problems. It is proved that all solutions of these nonlinear optimal control problems together assure asymptotic stability of the entire system. Then, in the framework of adaptive critics, the critic-only architecture is built to solve the Hamilton–Jacobi–Bellman equations associated with these solutions. The critic-only architecture is implemented via critic neural networks (NNs) with their weight vectors being tuned through an improved gradient descent method. A remarkable feature of the present gradient descent approach is that it simultaneously utilizes previously stored and instantaneous state data, which makes the persistence of excitation conditions relaxed. After that, with Lyapunov’s techniques being employed, asymptotic stability of the closed-loop auxiliary subsystems and uniform ultimate boundedness of the critic NNs’ weight estimation errors are demonstrated. Finally, simulations of an unmatched interconnected nonlinear plant are provided to validate the present decentralized control method.

In this paper, we consider a general class of nonlinear mixed discrete programming problems. By introducing continuous variables to replace the discrete variables, the problem is first transformed into an equivalent nonlinear continuous optimization problem subject to original constraints and additional linear and quadratic constraints. Then, an exact penalty function is employed to construct a sequence of unconstrained optimization problems, each of which can be solved effectively by unconstrained optimization techniques, such as conjugate gradient or quasi-Newton methods. It is shown that any local optimal solution of the unconstrained optimization problem is a local optimal solution of the transformed nonlinear constrained continuous optimization problem when the penalty parameter is sufficiently large. Numerical experiments are carried out to test the efficiency of the proposed method.