Output‐feedback Q‐learning for discrete‐time linear H∞ tracking control: A Stackelberg game approach

This article addresses the online reinforcement Q-learning algorithms to design H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking controller for unknown discrete-time linear systems. An augmented system composed of the original system and the command generator is constructed, and a discounted performance function is introduced to establish a discounted game algebraic Riccati equation (GARE). The existence conditions of a solution to the GARE are proposed and a lower bound is found for the discount factor to assure the stability of the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking control solution. The Q-function Bellman equation is then derived, based on which the reinforcement Q-learning algorithm is developed to learn the solution to H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking control problem without knowing the system dynamics. Both state-data-driven and output-data-driven reinforcement Q-learning algorithms toward finding the control policies are proposed. Unlike the value function approximation (VFA)-based approach, it is proved that the Q-learning scheme brings out no bias of solution to the Q-function Bellman equation under the probing noise satisfying the persistent excitation (PE) condition, and therefore, converges to the nominal discounted GARE solution. Moreover, the proposed output-data-driven method is more powerful than the state-data-driven method as it may not be available to completely measure the full system states in practical applications. A simulation example with a single-phase voltage-source UPS inverter is used to verify the effectiveness of the proposed Q-learning algorithms.

Optimal encryption strategy for cyber-physical systems against stealthy attacks with energy constraints: A Stackelberg game approach

The bipartite consensus (BC) issue for nonlinear multiagent systems (NMASs) with unknown system dynamics information is investigated in this article. Initially, the dynamics of NMASs are represented using the Takagi-Sugeno (T-S) fuzzy model. Subsequently, to achieve distributed control, a minmax game policy is introduced, where each agent aims to minimize its performance index while its neighbors attempt to maximize it. Consequently, the BC problem for NMASs is reformulated as a zero-sum game, transforming the controller design into solving a set of game algebraic Riccati equations (GAREs). To solve such equations, a novel scaling off-policy iteration (PI) algorithm is proposed. The key features of the proposed learning algorithm can be outlined in three main aspects: 1) during the learning process, the reliance on system dynamics is relaxed; 2) compared with the PI method, the requirement for initial admissible control policies is eliminated; and 3) a more rapid convergence speed is achieved than traditional value iteration. Finally, the effectiveness and advantages of the proposed method are validated through a simulation example and a series of comparative experiments.

A method is presented to solve two-person, single-stage, deterministic, Stackelberg games. The approach lies on the delineation of the inducible region. If the leader is extremely powerful, he can induce the follower to do whatever he desires. However, the leader in general can only induce the follower's behavior to certain extent due to limited information, control, etc. In other words, there exists a region inside which the leader can induce the follower's behavior. Thus, the best the leader can do is to minimize his cost inside this region. In this paper, we delineate precisely the inducible region, and obtain a leader's optimal strategy, if it exists. In case that a Stackelberg strategy does not exist, we show that an ?-Stackelberg strategy always exists.

Over the past decades, various theories and algorithms have been developed under the framework of Stackelberg games and part of these innovations have been fielded under the scenarios of national security defenses and wildlife protections. However, one of the remaining difficulties in the literature is that most of theoretical works assume full information of the payoff matrices, while in applications, the leader often has no prior knowledge about the follower’s payoff matrix, but may gain information about the follower’s utility function through repeated interactions. In this paper, we study the problem of learning the optimal leader strategy in Stackelberg (security) games and develop novel algorithms as well as new hardness results.

We study the problem of optimizing the performance of a system shared by selfish, noncooperative users. We consider the concrete setting of scheduling jobs on a set of shared machines with load-dependent latency functions specifying the length of time necessary to complete a job; we measure system performance by the total latency of the system.Assigning jobs according to the selfish interests of individual users (who wish to minimize only the latency that their own jobs experience) typically results in suboptimal system performance. However, in many systems of this type there is a mixture of “selfishly controlled” and “centrally controlled” jobs; as the assignment of centrally controlled jobs will influence the subsequent actions by selfish users, we aspire to contain the degradation in system performance due to selfish behavior by scheduling the centrally controlled jobs in the best possible way.We formulate this goal as an optimization problem via Stackelberg games, games in which one player acts a leader (here, the centralized authority interested in optimizing system performance) and the rest as followers (the selfish users). The problem is then to compute a strategy for the leader (a em Stackelberg strategy) that induces the followers to react in a way that (at least approximately) minimizes the total latency in the system.In this paper, we prove that it is NP-hard to compute the optimal Stackelberg strategy and present simple strategies with provable performance guarantees. More precisely, we give a simple algorithm that computes a strategy inducing a job assignment with total latency no more than a constant times that of the optimal assignment of all of the jobs; in the absence of centrally controlled jobs and a Stackelberg strategy, no result of this type is possible.

Optimal configuration and pricing strategies for electric-heat cloud energy storage: A Stackelberg game approach

With the explosive growth of indoor data traffic in forthcoming fifth generation cellular networks, it is imperative for mobile network operators to improve network coverage and capacity. Femtocells are widely recognized as a promising technology to address these demands. As femtocells are sold or loaned by a mobile network operator (MNO) to its residential or enterprise customers, MNOs usually employ refunding scheme to compensate the femtocell holders (FHs) providing indoor access to other subscribers by configuring the femtocell to operate in open or hybrid access mode. Due to the selfishness nature, competition between network operators as well as femtocell holders makes it challenging for operators to select appropriate FHs for trading access resources. This inspires us to develop an effective refunding framework, with aim to improve overall network resource utilization, through promoting FHs to make reasonable access permission for well-matched macro users. In this paper, we develop a two-stage auction–Stackelberg game (ASGF) framework for access permission in femtocell networks, where MNO and mobile virtual network operator lease access resources from multiple FHs. We first design an auction mechanism to determine the winner femtocell that fulfils the access request of macro users. We next formulate the access permission problem between the winner femtocell and operators as a Stackelberg game, and theoretically prove the existence of unique equilibrium. As a higher system payoff can be gained by improving individual players’ payoff in the game, each player can choose the best response to others’ action by implementing access permission, while avoiding solving a complicated optimization problem. Numerical results validate the effectiveness of our proposed ASGF based refunding framework and the overall network efficiency can be improved significantly.

Output feedback Q-learning for discrete-time linear zero-sum games with application to the H-infinity control

Cancer treatment faces significant challenges due to tumor adaptation, heterogeneity, and rapid resistance development, where sequential therapies often fail against the evolving dynamics of cancer cells. Traditional methods, such as fixed-dose regimens, overlook the adaptive, game-like interactions between oncologists and tumors, leading to suboptimal outcomes and limited personalization in precision medicine. To address these limitations, we introduce Nash-RL, an innovative framework that integrates game theory and reinforcement learning (RL) to enable optimized, evolutionarily informed cancer therapy. By modeling the oncologist-tumor interaction as a Stackelberg game, where the oncologist acts as an RL-trained leader optimizing strategies and the tumor as an evolutionary follower adapting to pressures, our approach anticipates resistance mechanisms and captures the sequential decision-making process inherent in adaptive treatments. Nash-RL combines key game-theoretic concepts, including Nash equilibrium, Stackelberg strategies, and evolutionary games, with multi-agent RL algorithms to develop robust, patient-specific treatment protocols. This integration not only enhances therapy efficacy by dynamically adjusting doses but also delays resistance development, advancing fields like adaptive therapy and precision oncology. We formulate the mathematical foundations, incorporating state-action-reward models and equilibrium computations, and outline computational implementation using deep Q-networks and policy gradient methods. Theoretical advantages over existing RL or game-theoretic approaches are analyzed, with proof-of-concept simulations demonstrating practical feasibility. Overall, Nash-RL provides a scalable foundation for sustainable strategies that collaborate with, rather than combat, cancer's adaptive nature, paving the way for intelligent, personalized oncology and potentially transforming clinical practice.

Solutions and properties of multi-stage stackelberg games

This paper will present an Approximate/Adaptive Dynamic Programming (ADP) algorithm for determining online the Nash equilibrium solution for the two-player zero-sum differential game with linear dynamics and infinite horizon quadratic cost. The algorithm is built around an iterative method that has been developed in the control engineering community for solving the continuous-time game algebraic Riccati equation (CT-GARE) that is underlying the game problem. We here show how the ADP techniques will enhance the capabilities of the offline method allowing an online solution without the requirement of complete knowledge of the system dynamics.

We investigate the tracking control problem for switched linear time-varying delays systems with stabilizable and unstabilizable subsystems. Sufficient conditions for the solvability of the tracking control problem are developed. The tracking control problem of a switched time-varying delays system with stabilizable and unstabilizable subsystems is solvable if the stabilizable and unstabilizable subsystems satisfy certain conditions and admissible switching law among them. Average dwell time approach and piecewise Lyapunov functional methods are utilized to the stability analysis and controller design. A simulation example shows the effectiveness of the proposed method.

Caching can effectively smooth the temporal traffic variability and decrease the redundant data transmission in mobile video delivery. In this paper, we consider a video caching system consisting of a video provider (VP), a mobile network operator (MNO) with a set of cache-enabled base stations (BSs), and multiple mobile users. The VP leases some popular videos to the MNO, while the MNO places these rented videos in local caches of its BSs to save expensive backhaul transmission cost. However, in such a two-sided market, these two entities are competing with each other for their own profit due to their opposite expectation on the video pricing. To address this, we model the competition between the two entities using the framework of Stackelberg games and propose a joint video pricing and cache placement strategy by considering the heterogeneity of video file sizes and exploiting the classic law of demand from the field of economics. The proposed optimization problem is able to jointly maximize the profit of the VP and the MNO by the optimal selection of the video pricing and the cache placement strategy given that price, for both noncooperative BS caching and cooperative BS caching cases. We then develop iterative algorithms based on dynamic programming and gradient ascent, respectively, for these two cases to find the Stackelberg equilibrium (SE). The simulation results further show that the proposed joint optimization formulation follows the law of demand in economics, and the proposed algorithms for both cases can efficiently converge to the SE point that jointly maximizes the profit for both the VP and the MNO.

Based on the framework of Stackelberg game, we analyze and compare the optimal channel choices for the manufacturer and the retailer under consignment contract, when the demand is stochastic with uniformly distributed shock in multiplicative and additive forms respectively. We find that the price elasticity, range of demand fluctuation and the revenue sharing ratio affect the optimal channel choices for the manufacturer and the retailer differently under different forms of stochastic demand shocks.