On a Problem of Optimal Stochastic Control with Incomplete Information

We consider a finite-capacity single-server queuing system in which the problems of capacity design and service quality control are integrated. The amount of time spent on servicing a customer is used as a measure of the quality of service provided. The expected reward from servicing is assumed to be a nondecreasing function of the service duration, while a linear holding cost is assumed for customers waiting in the system. We maximize the long-run average return per unit time by optimally controlling the quality of service to be provided as a function of the workload facing the server for different waiting-room capacities. We show that the optimal control policy for a given capacity is monotone and also examine the effect of varying the system capacity on the optimal control policy. Furthermore, we analyze the design problem of selecting an optimal capacity by assuming that an optimal service control policy will be followed for each system capacity.

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Dynamic control of a closed two-stage queueing network for outfitting process in shipbuilding

A generalized formulation of optimal control policy for physical stochastic optimal control of structures is proposed in this paper, which extends the classic forms of optimal control policies of passively controlled modalities, actively controlled modalities, semi-actively controlled modalities, and hybridly controlled modalities. This unified scheme features the simultaneous consideration of optimal control force and optimal placement of control devices. A new controllability index associated with the exceedance probability is developed, which characterizes system safety, system serviceability, system comfort, control device workability, and their trade-off. A sequential procedure with a minimum storey controllability index gradient is defined to efficiently search for the optimal topology of control devices. Furthermore, a performance function for parameters optimization is proposed, which exhibits the same practical accounts as the controllability index. The resolution procedure of the generalized optimal control policy is detailed, and two randomly base-excited structural systems are investigated for illustrative purposes, which consist of a ten-storey shear frame equipped with viscoelastic dampers and an eight-storey shear frame controlled by active tendons. Numerical results reveal that the proposed generalized optimal control policy can gain the maximum control effectiveness with the minimum control cost. The controlled system response is significantly reduced and becomes smoother along the height of the structure, which results in a more desirable structural performance. Using the design criterion, meanwhile, of the minimum storey controllability index gradient, the structural stochastic optimal control achieves the objective performance more efficiently than that of the maximum storey controllability index currently in use. Copyright © 2011 John Wiley & Sons, Ltd.

Time optimal control problems involving high dimensional systems are generally very difficult to solve. Instead of using suboptimal control based on reduced system, iterative dynamic programming is used directly to establish the time optimal control policy of a binary distillation column described by 11 ordinary differential equations. The resulting optimal control policy that is bang‐bang in nature was obtained very rapidly. The minimum time of 2236 seconds obtained here is substantially better than 4120 seconds obtained by a suboptimal control policy. Simulations show that the procedure may be used for on‐line optimal control of the distillation column, since the optimal control policy can be established very fast on a personal computer.

In this paper, a self-learning control scheme is proposed for the infinite horizon optimal control of affine nonlinear systems based on the action dependent heuristic dynamic programming algorithm. The policy iteration technique is introduced to derive the optimal control policy with feasibility and convergence analysis. It shows that the “greedy” control action for each state is uniquely existent, the learned control policy after each policy iteration is admissible, and the optimal control policy is able to be obtained. Two three-layer perceptron neural networks are employed to implement the scheme. The critic network is trained by a novel rule to conform to the Bellman equation, and the action network is trained to yield a better control policy. Both training processes alternate until the optimal control policy is achieved. Two simulation examples are provided to validate the effectiveness of the approach.

In this paper, We consider an uncertain Markov decision process (MDP) with a control cost and a linear temporal logic (LTL) control specification. We propose a reinforcement learning (RL) based method for design of an optimal control policy by which the controlled MDP satisfies the control specification with probability 1 and minimizes an expected discounted sum of the control costs. First, we construct a deterministic Rabin automaton (DRA) that accepts all and only infinite words satisfying the LTL control specification. Second, we construct a product MDP of the MDP and the DRA to represent a dynamic control policy that satisfies the control specification. Third, we modify the product MDP in order to apply RL to the design of an optimal control policy. The control action of the modified product MDP is a pair of a pattern and an action, where the pattern is a set of actions. Moreover, we introduce a reward that represents both the satisfaction of the control specification and the minimally restrictiveness of the pattern. Finally, we proposed an algorithm for design of an optimal control policy that consists of a sequential decision making of two steps. At the first decision making, we select a pattern that maximizes a discounted sum of the reward. At the second one, we select an action from the pattern selected at the first one such that it minimizes the expected discounted sum of the costs. Moreover, we consider an illustrative example to show that the proposed algorithm can obtain an optimal control policy.

Dynamic Dispatch and Centralized Relocation of Cars in Ride-Hailing Platforms

Integrating product and process design with optimal control: A case study of solvent recycling

This paper examines the robustness characteristics of optimal control policies derived under the assumption of rational expectations to alternative models of expectations formation and uncertainty about the natural rates of interest and unemployment. We assume that agents have imperfect knowledge about the precise structure of the economy and form expectations using a forecasting model that they continuously update based on incoming data. We also allow for central bank uncertainty regarding the natural rates of interest and unemployment. We find that the optimal control policy derived under the assumption of perfect knowledge about the structure of the economy can perform poorly when knowledge is imperfect. These problems are exacerbated by natural rate uncertainty, even when the central bank's estimates of natural rates are efficient. We show that the optimal control approach can be made more robust to the presence of imperfect knowledge by deemphasizing the stabilization of real economic activity and interest rates relative to inflation in the central bank loss function. That is, robustness to the presence of imperfect knowledge about the economy provides an incentive to employ a "conservative" central banker. We then examine two types of simple monetary policy rules from the literature that have been found to be robust to model misspecification in other contexts. We find that these policies are robust to the alternative models of learning that we study and natural rate uncertainty and outperform the optimal control policy and generally perform as well as the robust optimal control policy that places less weight on stabilizing economic activity and interest rates.

This paper presents a novel decentralized control strategy for a class of uncertain nonlinear large-scale systems with mismatched interconnections. First, it is shown that the decentralized controller for the overall system can be represented by an array of optimal control policies of auxiliary subsystems. Then, within the framework of adaptive dynamic programming, a simultaneous policy iteration (SPI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations associated with auxiliary subsystem optimal control policies. The convergence of the SPI algorithm is guaranteed by an equivalence relationship. To implement the present SPI algorithm, actor and critic neural networks are applied to approximate the optimal control policies and the optimal value functions, respectively. Meanwhile, both the least squares method and the Monte Carlo integration technique are employed to derive the unknown weight parameters. Furthermore, by using Lyapunov’s direct method, the overall system with the obtained decentralized controller is proved to be asymptotically stable. Finally, the effectiveness of the proposed decentralized control scheme is illustrated via simulations for nonlinear plants and unstable power systems.

Tangential hand velocity profiles of rapid human arm movements often appear as sequences of several bell-shaped acceleration-deceleration phases called submovements or movement units. This suggests how the nervous system might efficiently control a motor plant in the presence of noise and feedback delay. Another critical observation is that stochastic-ity in a motor control problem makes the optimal control policy essentially different from the optimal control policy for the deterministic case. We use a simplified dynamic model of an arm and address rapid aimed arm movements. We use reinforcement learning as a tool to approximate the optimal policy in the presence of noise and feedback delay. Using a simplified model we show that multiple submovements emerge as an optimal policy in the presence of noise and feedback delay. The optimal policy in this situation is to drive the arm's end point close to the target by one fast submovement and then apply a few slow submovementsto accurately drive the arm's end point into the target region. In our simulations, the controller sometimes generates corrective submovements before the initial fast submovement is completed, much like the predictive corrections observed in a number of psychophysical experiments.

In this paper, secure communication over a buffer-aided two-hop communication link, in which a source sends data to a destination with the aid of a trusted half-duplex relay node in the presence of a passive eavesdropper, is studied. It is assumed that there is no direct link between the source and the destination, and the buffer-aided relay forwards the information to the destination by employing the decode-and-forward (DF) scheme. Both the source and the relay transmissions are assumed to be vulnerable to the eavesdropper. The perfect channel side information (CSI) of the network is assumed to be available at the source and the relay. Initially, conventional relaying protocols where equal partition of the time for the reception and transmission of the relay is considered. Then, the optimal time fraction with the optimal power control policy that maximizes the secrecy throughput is identified. Subsequently, the optimal joint link selection and power control policy that maximizes the secrecy throughput is derived. Numerical results demonstrate that the joint link selection and power control scheme can significantly improve the achievable secrecy throughput of the buffer-aided two-hop communication system.

For the design of call admission control (CAC) policy in wireless networks, how to decrease the average system cost is a key issue. In this paper, we study the optimal admission policy for minimizing the system cost. By modeling the admission control problem into a Markov decision process (MDP) and analyzing the corresponding value function, we obtain some monotonicity properties of the optimal policy. These properties suggest that the optimal admission control policy for the bandwidth asymmetry wireless networks should have a threshold structure and the threshold specified for a class of calls may change with the system state. Due to the prohibitively high complexity for computing the thresholds in a system with large state space, we propose a heuristic CAC policy called call-rate- based dynamic threshold (CRDT) policy to approximate the theoretical optimal policy based on the insights we obtain from the modeling and the analytical study on the properties of the optimal policy. The CRDT policy is efficient and can be easily implemented. The numerical results show that the performance of average system cost of the proposed CRDT policy is close to that of the optimal policy from the MDP model and is better than that of some known existing CAC schemes, including those performing well in bandwidth asymmetry wireless networks.

This survey provides a brief overview on the control Lyapunov function (CLF) and control barrier function (CBF) for general nonlinear-affine control systems. The problem of control is formulated as an optimization problem where the optimal control policy is derived by solving a constrained quadratic programming (QP) problem. The CLF and CBF respectively characterize the stability objective and the safety objective for the nonlinear control systems. These objectives imply important properties including controllability, convergence, and robustness of control problems. Under this framework, optimal control corresponds to the minimal solution to a constrained QP problem. When uncertainties are explicitly considered, the setting of the CLF and CBF is proposed to study the input-to-state stability and input-to-state safety and to analyze the effect of disturbances. The recent theoretic progress and novel applications of CLF and CBF are systematically reviewed and discussed in this paper. Finally, we provide research directions that are significant for the advance of knowledge in this area.