Undiscounted Markov decision chains with partial information; an algorithm for computing a locally optimal periodic policy

A discrete-event system (DES) is described by a state-transition model. By disabling and enabling controllable events according to a rule based on partial information about the current state of the system, a static state-feedback controller synthesizes a predicate specifying states reachable from the initial state in the controlled system. Concepts of controllable predicate and observable predicate are introduced. It is shown that a predicate can be synthesized by a static state-feedback controller utilizing only partial state information if and only if the predicate is both controllable and observable. Some general results are presented for a state-variable model of a controlled DES.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

We consider a server serving a time-slotted queued system of multiple packet-based flows, where not more than one flow can be serviced in a single time slot. The flows have exogenous packet arrivals and time-varying service rates. At each time, the server can observe instantaneous service rates for only a subset of flows (selected from a fixed collection of observable subsets) before scheduling a flow in the subset for service. We are interested in queue length aware scheduling to keep the queues short. The limited availability of instantaneous service rate information requires the scheduler to make a careful choice of which subset of service rates to sample. We develop scheduling algorithms that use only partial service rate information from subsets of channels, and that minimize the likelihood of queue overflow in the system. Specifically, we present a new joint subset-sampling and scheduling algorithm called Max-Exp that uses only the current queue lengths to pick a subset of flows, and subsequently schedules a flow using the Exponential rule. When the collection of observable subsets is disjoint, we show that Max-Exp achieves the best exponential decay rate, among all scheduling algorithms that base their decision on the current (or any finite past history of) system state, of the tail of the longest queue. To accomplish this, we employ novel analytical techniques for studying the performance of scheduling algorithms using partial state, which may be of independent interest. These include new sample-path large deviations results for processes obtained by non-random, predictable sampling of sequences of independent and identically distributed random variables. A consequence of these results is that scheduling with partial state information yields a rate function significantly different from scheduling with full channel information. In the special case when the observable subsets are singleton flows, i.e., when there is effectively no a priori channel state information, Max-Exp reduces to simply serving the flow with the longest queue; thus, our results show that to always serve the longest queue in the absence of any channel state information is large deviations optimal.

A channel model for channels with unknown state and partial state information available causally at the decoder is proposed. The model is based on arbitrarily varying channels (AVCs), and the partial state information is characterized by allowing the decoder to know a subset in which the average empirical channel must lie. A randomized rateless coding scheme for these channels is presented that achieves rates near that if the channel quality were known in advance under the assumption that the average channel can be estimated accurately. The same coding scheme can use used to multicast a common message to several receivers with different state sequences.

We consider the problem of optimal probing of communication channel at the encoder for discrete memoryless channels (DMC) with i.i.d. states. The encoder generates an action sequence dependent on the message with which it probes the channel and hence acquires partial state information. This is further used causally or non-causally to generate channel input symbols while decoder may or may not have partial channel state information. We call the maximum achievable rate for such systems as the `Probing Capacity'. In this paper we characterize this capacity when channel inputs are allowed to depend causally or non-causally on partially observed state sequence.We motivate the problem by addressing the issue of characterizing the essential trade-off between the capacity and the cost `to observe or not to observe' the channel states at the encoder.

We consider transmission over the ergodic fading multiple-antenna broadcast (MIMO-BC) channel with partial channel state information at the transmitter and full information at the receiver. Over the equivalent non-fading channel, capacity has recently been shown to be achievable using transmission schemes that were designed for the "dirty paper" channel. We focus on a similar "fading paper" model. The evaluation of the fading paper capacity is difficult to obtain. We confine ourselves to the linear-assignment capacity, which we define, and use convex analysis methods to prove that its maximizing distribution is Gaussian. We compare our fading-paper transmission to an application of dirty paper coding that ignores the partial state information and assumes the channel is fixed at the average fade. We show that a gain is easily achieved by appropriately exploiting the information. We also consider a cooperative upper bound on the sum-rate capacity as suggested by Sato. We present a numeric example that indicates that our scheme is capable of realizing much of this upper bound.

Optimal policy with general signal extraction

We are motivated to improve the performance of Vertical Bell Laboratories Layered Space-Time (V-BLAST) systems by applying link adaptation techniques based on partial channel state information (CSI) feedback. In our scheme, the partial channel state information is the second order statistics that are always available in practical applications. We derive the criteria that decide the optimal transmission policy based on minimum bit error probability. Simulations show that our scheme provides considerable performance gain over traditional V-BLAST transmission scheme. Since the second-order channel statistics varies much more slowly than instantaneous channel realization, the proposed technique would have practical applications in downlink multi-input multi-output (MIMO) systems that exhibit fading correlations between antennas.

This paper focuses on controller and observer design for the longitudinal model of an air-breathing hypersonic vehicle (AHV) subject to actuator faults and limited measurements of the states. The feedback linearization method is firstly employed for a modified AHV model with actuator faults, and dynamic effect caused by the actuator faults on the linearized model is analyzed. Based on full state information, an adaptive controller is designed using the Lyapunov method, which guarantees reference command tracking of the AHV under actuator faults. Next, to estimate the unmeasurable states used in the adaptive controller, a sliding observer is designed based on the sliding control method and the Filippov’s construction of the equivalent dynamics (FCED). Finally, the adaptive controller is combined with the sliding observer to generate the observer-based adaptive controller, which relies only on partial state information. Simulations demonstrate that the observer-based adaptive controller achieves desired tracking performance and good robustness in the presence of actuator faults.

We investigate the capacity of a multiple access channel with cooperating encoders where partial state information is known to each encoder in a non-causal way and full state information is known to the decoder. The cooperation between the encoders has a two-fold purpose: to generate empirical state coordination between the encoders, and to share information about the private messages that each encoder has. For two-way cooperation, this two-fold purpose is achieved by double-binning, where the first layer of binning is used to generate the state coordination similarly to the two-way source coding, and the second layer of binning is used to transmit information about the private messages. The complete result provides the framework and perspective for addressing a complex level of cooperation that mixes states and messages in an optimal way.

A stochastic adaptive control problem where complete state information is not available to the controller is considered. The system is modeled as a finite stochastic automaton. These models are a slight generalization of the more common partially observable controlled Markov chain models.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

We study joint replenishment and clearance sales of perishable goods under a general finite lifetime and a last-in-first-out (LIFO) issuing rule, a problem common in retailing. We show that the optimal policies can be characterized by two thresholds for each age group of inventory: a lower one and a higher one. For an age group of inventory with a remaining lifetime of two periods or longer, if its inventory level is below its lower threshold, then there is no clearance sales; if it is above its higher threshold, then it will be cleared down to the higher threshold. The optimal policy for the age group of inventory with a one-period remaining lifetime is different. Clearance sales may occur if its inventory level is above its higher threshold or below its lower threshold. The phenomenon that a clearance sale happens when the inventory is low is driven by the need to segregate the newest inventory from the oldest inventory and is unique to the LIFO issuing rule. The optimal policy requires a full inventory record of every age group and its computation is challenging. We consider two myopic heuristics that require only partial information. The first requires only the information about the total inventory and the second requires the information about the total inventory as well as the information about the inventory with a one-period remaining lifetime. Our numerical studies show that the second outperforms the first significantly and its performance is consistently very close to that of the optimal policy.

Robust and risk sensitive control of discrete time, finite state systems on an infinite horizon is considered, with either state or output feedback. The solution of the state feedback robust control problem is characterized in terms of the value of an average cost dynamic game. The risk sensitive stochastic optimal control problem is solved using the policy iteration algorithm, and the optimal rate is expressed in terms of the value of a stochastic dynamic game with average cost per unit time criterion. By taking a small noise limit a deterministic dynamic game is obtained, which is closely related to the robust control problem. For the problem with output feedback (partial state information) the analysis depends on introducing appropriate information states for the risk sensitive and robust control problems.

THE ELEVATOR DISPATCHING PROBLEM: HYBRID SYSTEM MODELING AND RECEDING HORIZON CONTROL

This paper presents a solution to a robust optimal regulation problem for a nonlinear polynomial system affected by parametric and matched uncertainties, which is based only on partial state information. The parameters describing the dynamics of the nonlinear polynomial plant depend on a vector of unknown parameters, which belongs to a finite parametric set, and the application of a certain control input is associated with the worst or least favorable value of the unknown parameter. A high-order sliding mode state reconstructor is designed for the nonlinear plant in such a way that the previously designed control can be applied for a system with incomplete information. Additionally, the matched uncertainty is also compensated by means of the same output-based regulator. The obtained algorithm is applied to control a mechanical pendulum of the third order, successfully verifying the effectiveness of the developed approach.