From drinking philosophers to asynchronous path-following robots

This article studies a distributed minmax strategy for multiplayer games and develops reinforcement learning (RL) algorithms to solve it. The proposed minmax strategy is distributed, in the sense that it finds each player's optimal control policy without knowing all the other players' policies. Each player obtains its distributed control policy by solving a distributed algebraic Riccati equation in a multiplayer noncooperative game. This policy is found against the worst policies of all the other players. We guarantee the existence of distributed minmax solutions and study their L2 and asymptotic stabilities. Under mild conditions, the resulting minmax control policies are shown to improve robust gain and phase margins of multiplayer systems compared to the standard linear-quadratic regulator controller. Distributed minmax solutions are found using both model-based policy iteration and data-driven off-policy RL algorithms. Simulation examples verify the proposed formulation and its computational efficiency over the nondistributed Nash solutions.

Stability and robustness analysis of minmax solutions for differential graphical games

In this article, a distributed resource allocation problem for second‐order multi‐agent systems over undirected switching topologies is investigated. For the distributed resource allocation problem, most of the existing algorithms depend on the gradient of cost function with specific expression, which may be unknown to agents. To deal with unknown cost functions, the extremum seeking control is introduced in the designed distributed resource allocation algorithm to estimate the gradient of the cost function only by the value of the cost. Besides, an estimator is proposed to estimate the optimal Lagrange multiplier associated with the equality constraint. The communication graph is supposed to be switched among a set of undirected subgraphs, which may be connected or unconnected. For undirected and connected switching graphs, the average dwell‐time constraint is used to describe the switching frequency. For the undirected graphs, containing connected and unconnected ones, both the average dwell‐time and the time‐ratio constraints are used to model the switching signal. For the two cases, the semi‐globally practically asymptotical stability of the designed algorithm is established. The obtained results are verified by an example.

The symmetric and weighted Shapley values for cooperative n-person games are studied. Using the semi-tensor product of matrices, it is first shown that a characteristic function can be expressed as a pseudo-Boolean function. Then, two simple matrix formulas are obtained for calculating the symmetric and weighted Shapley values. Finally, using these new formulas, a design technique for the agents’ payoff functions in distributed resource allocation problems is proposed. It is possible to design payoff functions with the weighted Shapley value by the nonsymmetric weights defined on the players, thus ensuring that the optimal allocation is a pure Nash equilibrium. Practical examples are presented to illustrate the theoretical results.

Energy-efficiency, high transmission data rates and Quality of Service (QoS) awareness are the major challenges for resource management in the uplink of Visible Light Communication Personal Area Networks (VPANs). This paper investigates the problem of Optical Access Point (OAP) selection and resource allocation in the uplink of VPANs under two different transmission techniques, namely Orthogonal Frequency Division Multiple Access (OFDMA) and Non-Orthogonal Multiple Access (NOMA). Each user is associated with a generic utility function, which represents his perceived satisfaction with respect to the overall resource allocation problem. OAP selection adopts Maximum Gain Selection (MGS) policy, i.e. users select an OAP to connect to based on the highest path gain. A distributed resource allocation problem in VPANs is formulated and solved as an optimization problem. Following this analysis, a decentralized iterative and low-complexity algorithm for determining OAP selection and resource allocation is proposed, while the overall approach’s efficiency is illustrated via modeling and simulation, highlighting and assessing the advantages and drawbacks of each adopted transmission technique, i.e. OFDMA and NOMA.

In this paper, a distributed resource allocation problem is investigated based on a continuous-time algorithm in the presence of time-varying communication delays. The considered delays are allowed to take both "large" and "small" delay values in an alternating manner. The "small" delays refer to a class of delays that guarantee the exponential convergence to the optimal solution; otherwise, they are called "large" delays. The existence of "large" delays may affect the behavior of the optimal allocation solution, and the existing algorithms with delays may not be applicable to this kind of "large" delays. To address "large" delay cases, a switched algorithm is first proposed based on switching techniques, and then, such an algorithm is modeled as switched delay systems. Second, by adopting the Lyapunov functional theory, the upper bound of delays is first derived to guarantee the exponential convergence to the optimal solution under "small" delays, and then, a delay-dependent sufficient condition is obtained to ensure exponential convergence of the proposed algorithm based on some restrictions on "large" delays. Finally, we adopt an IEEE 118-bus practical example to show the effectiveness of the proposed algorithm.

In this paper, the distributed resource allocation problem over undirected networks, considering box constraints and demand–supply constraints, is studied. Furthermore, each agent has one private cost function that is only known by itself. The aim is to minimize the sum of all local cost functions in a distributed manner while meeting the constraints. To solve the dispatch problem, a continuous‐time distributed algorithm with fixed step‐size is proposed. After that, a modified distributed algorithm of discrete‐time multi‐agent system is provided. Moreover, the proposed algorithms obtain the global optimal solution with an exponential convergence speed by the conventional Lyapunov methods. Finally, several simulations of IEEE‐14 bus are presented to validate and illustrate the proposed designed algorithm.

This paper proposes distributed algorithms for control and coordination of loads and distributed energy resources (DERs) in distribution networks. These algorithms are relevant for load curtailment control in demand response programs, and also for coordination of DERs for provision of ancillary services. Both the distributed load-curtailment and DER coordination problems can be cast as distributed resource allocation problems with constraints on resource capacity. We focus on linear iterative algorithms in which each resource j maintains a set of values that is updated to be a weighted linear combination of the resource's own previous set of values and the previous sets of values of its neighboring resources. This set of values can be used by each node to determine its own contribution to load curtailment or to resource request.

The economic management of a microgrid can greatly benefit from energy storage systems (ESSs), which may act as virtual load deferral systems to take advantage of the fluctuations of energy prices and accommodate for demand-production mismatches caused by the scarce predictability of renewable sources. In a distributed energy management scenario, an ESS may serve multiple users, a setting which calls for the development of suitable resource allocation policies for the storage capacity. In particular, distributed control policies are of interest, where each user operates independently with the least exchange of information with the other users. A methodology is developed in the paper for such purpose, based on an iterative resource allocation mechanism, realized by means of a negotiation process among users, resembling stock exchange dynamics. The resulting distributed strategy for the management of the shared resource comes close to optimality at a low computational cost, which is affordable in large scale practical applications. It is also robust to communication failures between users.

In this paper, the multiagent pursuit-evasion (MPE) games are solved in order to obtain optimal strategic policies for all players. In these games, multiple pursuers attempt to intercept multiple evaders who try to avoid capture. A graph-theoretic approach is employed to study the interactions of the agents with limited sensing capabilities, such that distributed control policies are obtained for every agent. Furthermore, the minimization of performance indices associated with the goals of the agents is guaranteed. Nash equilibrium among the players is obtained by means of optimal policies that use the solutions of the Hamilton–Jacobi–Isaacs (HJI) equations of the game. Minmax strategies are also proposed to guarantee a security-level performance when the solutions of the HJI equations for Nash equilibrium do not exist. Scenarios for finite-time capture and for asymptotic rendezvous are analyzed, and emergent behaviors are obtained by means of modifications of the proposed general-case performance indices. The containment control results are shown to be special cases of the solutions of the MPE games.

In this paper, differential games with incomplete information, or Bayesian games, are formulated for multiagent continuous-time dynamical systems in a communication graph. These new Bayesian graphical games model the situation where the agents are uncertain about their actual payoff functions and must employ the evidence observed from the behavior of other agents to improve their estimation of the real setting of their environment. Two different solutions are proposed for the game. First, the agents play their best response strategies with respect to the policies of their neighbors; then, the agents prepare themselves for the worst-case neighbor policies. A tight relationship between the beliefs of an agent and its distributed control policy is established. Two belief update methodologies are proposed for the agents to review their strategies using information obtained from the graph topology.

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

In this paper, a self-learning control scheme is proposed for the infinite horizon optimal control of affine nonlinear systems based on the action dependent heuristic dynamic programming algorithm. The policy iteration technique is introduced to derive the optimal control policy with feasibility and convergence analysis. It shows that the “greedy” control action for each state is uniquely existent, the learned control policy after each policy iteration is admissible, and the optimal control policy is able to be obtained. Two three-layer perceptron neural networks are employed to implement the scheme. The critic network is trained by a novel rule to conform to the Bellman equation, and the action network is trained to yield a better control policy. Both training processes alternate until the optimal control policy is achieved. Two simulation examples are provided to validate the effectiveness of the approach.

An incremental iterative Q-learning algorithm (IIQLA) is proposed to tackle the optimal secure control problem for cyber-physical systems under false data injection attacks. Within a zero-sum game framework, the secure control problem is transformed into solving an iterative algebraic Riccati equation. To derive the optimal secure control policy from the equation, the IIQLA is developed, which does not require prior knowledge of the system dynamics. This algorithm utilizes two auxiliary variables to separate the behavior policy and the target policy, thereby improving the exploration of data. As a result, the devised control policies are more conservative, leading to solutions that are closer to the optimal policy. Moreover, by introducing an adaptive learning rate, the proposed IIQLA can accelerate the convergence speed and decrease the number of required iterations, thus alleviating the computation burden. The convergence of the proposed IIQLA with different learning rates is also analyzed. In addition, the closed-loop system is guaranteed to be asymptotically stable, and the exploration noise does not introduce bias into the optimal policies. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed approach.

Temporal logic and model-checking are useful theoretical tools for specifying complex goals at the task level and formally verifying the performance of control policies. We are interested in tasks that involve constraints on real-valued energy resources. In particular, autonomous gliding aircraft gain energy in the form of altitude by exploiting wind currents and must maintain altitude within some range during motion planning. We propose an extension to probabilistic computation tree logic that expresses such real-valued resource threshold constraints, and present model-checking algorithms that evaluate a piecewise control policy with respect to a formal specification and hard or soft performance guarantees. We validate this approach through simulated examples of motion planning among obstacles for an autonomous thermal glider. Our results demonstrate probabilistic performance guarantees on the ability of the glider to complete its task, following a given piecewise control policy, without knowing the exact path of the glider in advance.