Optimal control for systems with deterministic production cycles

This paper deals with the optimal control of a one-machine two-product manufacturing system with setup changes, operating in a continuous time dynamic environment. The system is deterministic. When production is switched from one product to the other, a known constant setup time and a setup cost are incurred. Each product has specified constant processing time and constant demand rate, as well as an infinite supply of raw material. The problem is formulated as a feedback control problem. The objective is to minimize the total backlog, inventory and setup costs incurred over a finite horizon. The optimal solution provides the optimal production rate and setup switching epochs as a function of the state of the system (backlog and inventory levels). For the steady state, the optimal cyclic schedule is determined. To solve the transient case, the system's state space is partitioned into mutually exclusive regions such that with each region, the optimal control policy is determined analytically.

In this paper, a self-learning control scheme is proposed for the infinite horizon optimal control of affine nonlinear systems based on the action dependent heuristic dynamic programming algorithm. The policy iteration technique is introduced to derive the optimal control policy with feasibility and convergence analysis. It shows that the “greedy” control action for each state is uniquely existent, the learned control policy after each policy iteration is admissible, and the optimal control policy is able to be obtained. Two three-layer perceptron neural networks are employed to implement the scheme. The critic network is trained by a novel rule to conform to the Bellman equation, and the action network is trained to yield a better control policy. Both training processes alternate until the optimal control policy is achieved. Two simulation examples are provided to validate the effectiveness of the approach.

Optimal control of production-dependent failure-prone manufacturing systems

We consider a finite-capacity single-server queuing system in which the problems of capacity design and service quality control are integrated. The amount of time spent on servicing a customer is used as a measure of the quality of service provided. The expected reward from servicing is assumed to be a nondecreasing function of the service duration, while a linear holding cost is assumed for customers waiting in the system. We maximize the long-run average return per unit time by optimally controlling the quality of service to be provided as a function of the workload facing the server for different waiting-room capacities. We show that the optimal control policy for a given capacity is monotone and also examine the effect of varying the system capacity on the optimal control policy. Furthermore, we analyze the design problem of selecting an optimal capacity by assuming that an optimal service control policy will be followed for each system capacity.

An incremental iterative Q-learning algorithm (IIQLA) is proposed to tackle the optimal secure control problem for cyber-physical systems under false data injection attacks. Within a zero-sum game framework, the secure control problem is transformed into solving an iterative algebraic Riccati equation. To derive the optimal secure control policy from the equation, the IIQLA is developed, which does not require prior knowledge of the system dynamics. This algorithm utilizes two auxiliary variables to separate the behavior policy and the target policy, thereby improving the exploration of data. As a result, the devised control policies are more conservative, leading to solutions that are closer to the optimal policy. Moreover, by introducing an adaptive learning rate, the proposed IIQLA can accelerate the convergence speed and decrease the number of required iterations, thus alleviating the computation burden. The convergence of the proposed IIQLA with different learning rates is also analyzed. In addition, the closed-loop system is guaranteed to be asymptotically stable, and the exploration noise does not introduce bias into the optimal policies. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed approach.

This paper studies optimal power control policies that maximize the achievable rates of underlay cognitive radio systems with arbitrary input distributions under both peak/average transmit power and peak/average interference power constraints for general fading distributions. In particular, optimal power adaptation schemes are formulated and low-complexity optimal power control algorithms are proposed. Additionally, simpler approximations of optimal power control policies in the low-power regime are determined. By considering gamma distributed channel power gains of the interference link between the secondary transmitter and the primary receiver and of the transmission link between the secondary transmitter and the secondary receiver, closed-form expressions for the maximum achievable rate attained with optimal power control in the low-power regime are provided. Through numerical results, the impact of the fading severity of both interference and transmission links and transmit power and interference power constraints on the maximum achievable rate of the cognitive user for different practical constellations and Gaussian signals are investigated.

Reduction of energy consumption has attracted much attention of the researchers and practitioners in the field of sustainable manufacturing. However, the issue of integrated optimal production and energy control for sustainable manufacturing systems has not yet been investigated sufficiently. In this paper, we study such a problem for a single machine and single product-type manufacturing system, with the objective of minimizing the sum of production cost and energy consumption cost. To solve this problem, a Markov decision process model is developed, based on which, we obtain the integrated optimal production and energy control policy by the policy iteration method. Furthermore, we reveal the structural characteristics of optimal control policy and analyze the influence of system parameters on it by comparing different cases.

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

This chapter obtains optimal solutions for encoders and controllers under quadratic cost functions for linear Gaussian systems controlled over Gaussian channels, proving also the existence of optimal solutions. Furthermore, the chapter identifies conditions under which optimal coding and control policies are linear. A large class of network settings where optimal policies are non-linear is identified.

Learning-based optimal cooperative control using edge-based triggering strategy

A generalized formulation of optimal control policy for physical stochastic optimal control of structures is proposed in this paper, which extends the classic forms of optimal control policies of passively controlled modalities, actively controlled modalities, semi-actively controlled modalities, and hybridly controlled modalities. This unified scheme features the simultaneous consideration of optimal control force and optimal placement of control devices. A new controllability index associated with the exceedance probability is developed, which characterizes system safety, system serviceability, system comfort, control device workability, and their trade-off. A sequential procedure with a minimum storey controllability index gradient is defined to efficiently search for the optimal topology of control devices. Furthermore, a performance function for parameters optimization is proposed, which exhibits the same practical accounts as the controllability index. The resolution procedure of the generalized optimal control policy is detailed, and two randomly base-excited structural systems are investigated for illustrative purposes, which consist of a ten-storey shear frame equipped with viscoelastic dampers and an eight-storey shear frame controlled by active tendons. Numerical results reveal that the proposed generalized optimal control policy can gain the maximum control effectiveness with the minimum control cost. The controlled system response is significantly reduced and becomes smoother along the height of the structure, which results in a more desirable structural performance. Using the design criterion, meanwhile, of the minimum storey controllability index gradient, the structural stochastic optimal control achieves the objective performance more efficiently than that of the maximum storey controllability index currently in use. Copyright © 2011 John Wiley & Sons, Ltd.

Dynamic control of a closed two-stage queueing network for outfitting process in shipbuilding

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Integrating product and process design with optimal control: A case study of solvent recycling

This paper examines the robustness characteristics of optimal control policies derived under the assumption of rational expectations to alternative models of expectations formation and uncertainty about the natural rates of interest and unemployment. We assume that agents have imperfect knowledge about the precise structure of the economy and form expectations using a forecasting model that they continuously update based on incoming data. We also allow for central bank uncertainty regarding the natural rates of interest and unemployment. We find that the optimal control policy derived under the assumption of perfect knowledge about the structure of the economy can perform poorly when knowledge is imperfect. These problems are exacerbated by natural rate uncertainty, even when the central bank's estimates of natural rates are efficient. We show that the optimal control approach can be made more robust to the presence of imperfect knowledge by deemphasizing the stabilization of real economic activity and interest rates relative to inflation in the central bank loss function. That is, robustness to the presence of imperfect knowledge about the economy provides an incentive to employ a "conservative" central banker. We then examine two types of simple monetary policy rules from the literature that have been found to be robust to model misspecification in other contexts. We find that these policies are robust to the alternative models of learning that we study and natural rate uncertainty and outperform the optimal control policy and generally perform as well as the robust optimal control policy that places less weight on stabilizing economic activity and interest rates.