Optimal and Efficient Stochastic Motion Planning in Partially-Known Environments

Dynamic control of a closed two-stage queueing network for outfitting process in shipbuilding

We consider a finite-capacity single-server queuing system in which the problems of capacity design and service quality control are integrated. The amount of time spent on servicing a customer is used as a measure of the quality of service provided. The expected reward from servicing is assumed to be a nondecreasing function of the service duration, while a linear holding cost is assumed for customers waiting in the system. We maximize the long-run average return per unit time by optimally controlling the quality of service to be provided as a function of the workload facing the server for different waiting-room capacities. We show that the optimal control policy for a given capacity is monotone and also examine the effect of varying the system capacity on the optimal control policy. Furthermore, we analyze the design problem of selecting an optimal capacity by assuming that an optimal service control policy will be followed for each system capacity.

In this paper, We consider an uncertain Markov decision process (MDP) with a control cost and a linear temporal logic (LTL) control specification. We propose a reinforcement learning (RL) based method for design of an optimal control policy by which the controlled MDP satisfies the control specification with probability 1 and minimizes an expected discounted sum of the control costs. First, we construct a deterministic Rabin automaton (DRA) that accepts all and only infinite words satisfying the LTL control specification. Second, we construct a product MDP of the MDP and the DRA to represent a dynamic control policy that satisfies the control specification. Third, we modify the product MDP in order to apply RL to the design of an optimal control policy. The control action of the modified product MDP is a pair of a pattern and an action, where the pattern is a set of actions. Moreover, we introduce a reward that represents both the satisfaction of the control specification and the minimally restrictiveness of the pattern. Finally, we proposed an algorithm for design of an optimal control policy that consists of a sequential decision making of two steps. At the first decision making, we select a pattern that maximizes a discounted sum of the reward. At the second one, we select an action from the pattern selected at the first one such that it minimizes the expected discounted sum of the costs. Moreover, we consider an illustrative example to show that the proposed algorithm can obtain an optimal control policy.

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

In this paper, a self-learning control scheme is proposed for the infinite horizon optimal control of affine nonlinear systems based on the action dependent heuristic dynamic programming algorithm. The policy iteration technique is introduced to derive the optimal control policy with feasibility and convergence analysis. It shows that the “greedy” control action for each state is uniquely existent, the learned control policy after each policy iteration is admissible, and the optimal control policy is able to be obtained. Two three-layer perceptron neural networks are employed to implement the scheme. The critic network is trained by a novel rule to conform to the Bellman equation, and the action network is trained to yield a better control policy. Both training processes alternate until the optimal control policy is achieved. Two simulation examples are provided to validate the effectiveness of the approach.

A generalized formulation of optimal control policy for physical stochastic optimal control of structures is proposed in this paper, which extends the classic forms of optimal control policies of passively controlled modalities, actively controlled modalities, semi-actively controlled modalities, and hybridly controlled modalities. This unified scheme features the simultaneous consideration of optimal control force and optimal placement of control devices. A new controllability index associated with the exceedance probability is developed, which characterizes system safety, system serviceability, system comfort, control device workability, and their trade-off. A sequential procedure with a minimum storey controllability index gradient is defined to efficiently search for the optimal topology of control devices. Furthermore, a performance function for parameters optimization is proposed, which exhibits the same practical accounts as the controllability index. The resolution procedure of the generalized optimal control policy is detailed, and two randomly base-excited structural systems are investigated for illustrative purposes, which consist of a ten-storey shear frame equipped with viscoelastic dampers and an eight-storey shear frame controlled by active tendons. Numerical results reveal that the proposed generalized optimal control policy can gain the maximum control effectiveness with the minimum control cost. The controlled system response is significantly reduced and becomes smoother along the height of the structure, which results in a more desirable structural performance. Using the design criterion, meanwhile, of the minimum storey controllability index gradient, the structural stochastic optimal control achieves the objective performance more efficiently than that of the maximum storey controllability index currently in use. Copyright © 2011 John Wiley & Sons, Ltd.

Dynamic Dispatch and Centralized Relocation of Cars in Ride-Hailing Platforms

For the design of call admission control (CAC) policy in wireless networks, how to decrease the average system cost is a key issue. In this paper, we study the optimal admission policy for minimizing the system cost. By modeling the admission control problem into a Markov decision process (MDP) and analyzing the corresponding value function, we obtain some monotonicity properties of the optimal policy. These properties suggest that the optimal admission control policy for the bandwidth asymmetry wireless networks should have a threshold structure and the threshold specified for a class of calls may change with the system state. Due to the prohibitively high complexity for computing the thresholds in a system with large state space, we propose a heuristic CAC policy called call-rate- based dynamic threshold (CRDT) policy to approximate the theoretical optimal policy based on the insights we obtain from the modeling and the analytical study on the properties of the optimal policy. The CRDT policy is efficient and can be easily implemented. The numerical results show that the performance of average system cost of the proposed CRDT policy is close to that of the optimal policy from the MDP model and is better than that of some known existing CAC schemes, including those performing well in bandwidth asymmetry wireless networks.

Dynamic power management (DPM) plays a significant role to save the power consumption effectively in both design and operational phases of computer-based systems. Especially, it is well known that the state-dependent control policy by monitoring the energy state in each component or the whole system is efficient for power saving in server systems whose system state such as the number of transaction requests can be completely observed. In this paper we consider an optimal power-aware design in a cluster system and formulate the DPM problem by means of a Markov decision process. We derive the dynamic programming equation for the optimal control policy which maximizes the expected reward per unit electrical power, called the power effectiveness, and give a policy iteration algorithm to determine the optimal control policy sequentially. In a numerical example, we consider a cluster system with two service nodes and describe the arrival stream of the transaction request process by a Markov modulated Poisson process. It is shown that the optimal control policy maximizing the power effectiveness is given in the form of decision table.

Time optimal control problems involving high dimensional systems are generally very difficult to solve. Instead of using suboptimal control based on reduced system, iterative dynamic programming is used directly to establish the time optimal control policy of a binary distillation column described by 11 ordinary differential equations. The resulting optimal control policy that is bang‐bang in nature was obtained very rapidly. The minimum time of 2236 seconds obtained here is substantially better than 4120 seconds obtained by a suboptimal control policy. Simulations show that the procedure may be used for on‐line optimal control of the distillation column, since the optimal control policy can be established very fast on a personal computer.

Dynamic production control in parallel processing systems under process queue time constraints

This paper investigates an optimal decentralized control problem for a system with B-matrices dependent on stochastic parameters. It is assumed that these parameters are independent in time and available locally to each controller. The objective is to find a decentralized state feedback control policy that minimizes a multi step quadratic cost function. We first solve this problem for the case of one time step and show that the optimal policy can be reached through iterating the best responses of each player. For the optimal multiple step problem, a dynamic programming approach is employed while using the result of the one step control at each step. Finally in the infinite horizon case, we provide sufficient conditions under which the optimal control policy is unique and stabilizes the system in a mean square sense.

This paper studies the optimal policy for joint control of admission, routing, service, and jockeying in a queueing system consisting of two exponential servers in parallel. Jobs arrive according to a Poisson process. Upon each arrival, an admission/routing decision is made, and the accepted job is routed to one of the two servers with each being associated with a queue. After each service completion, the servers have an option of serving a job from its own queue, serving a jockeying job from another queue, or staying idle. The system performance is inclusive of the revenues from accepted jobs, the costs of holding jobs in queues, the service costs and the job jockeying costs. To maximize the total expected discounted return, we formulate a Markov decision process (MDP) model for this system. The value iteration method is employed to characterize the optimal policy as a hedging point policy. Numerical studies verify the structure of the hedging point policy which is convenient for implementing control actions in practice.

This paper presents a hierarchical approach for controlling the spread of an epidemic disease. The approach consists of a three-layer architecture where a set of two-layer multiple social networks is governed by a (third) top-layer consisting of an optimal control policy. Each of the two-layer social networks is modeled by a microscopic Markov chain. On top of all the two-layer networks is an optimal control policy that has been developed by using an underlying Markov Decision Process (MDP) model. Mathematical models pertaining to the top-level MDP as well as two-layer microscopic Markov chains have been presented. Practical implementation methodology using the proposed models has also been discussed along with a numerical example. The results in the numerical example illustrate the control of an epidemic using the optimal policy. Directions for further research and characterization of the optimal policy have also been discussed with the help of the same numerical example.Article HighlightsAn optimal approach for controlling the spread of an epidemic infection.The approach is able to model the uncertainties involved in the problem.The approach is able to cater for the underlying social network.

In this paper, secure communication over a buffer-aided two-hop communication link, in which a source sends data to a destination with the aid of a trusted half-duplex relay node in the presence of a passive eavesdropper, is studied. It is assumed that there is no direct link between the source and the destination, and the buffer-aided relay forwards the information to the destination by employing the decode-and-forward (DF) scheme. Both the source and the relay transmissions are assumed to be vulnerable to the eavesdropper. The perfect channel side information (CSI) of the network is assumed to be available at the source and the relay. Initially, conventional relaying protocols where equal partition of the time for the reception and transmission of the relay is considered. Then, the optimal time fraction with the optimal power control policy that maximizes the secrecy throughput is identified. Subsequently, the optimal joint link selection and power control policy that maximizes the secrecy throughput is derived. Numerical results demonstrate that the joint link selection and power control scheme can significantly improve the achievable secrecy throughput of the buffer-aided two-hop communication system.