Application of dynamic programming to priority assignment in a class of queueing systems with impatient customers

A state-space process-control model involving adjustment error and deterministic drift of the process mean is presented. The optimal adjustment policy is developed by dynamic programming. This policy calls for a particular adjustment when a Kalman-filter estimator is outside a deadband defined by upper and lower action limits. The effects of adjustment cost, adjustment variance, and drift rate on the optimal policy are discussed. The optimal adjustment policy is computed for a real machining process, and a simulation study is presented that compares the optimal policy to two sensible suboptimal policies.

An Optimal Control Problem with State Vector Measurement Errors

Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal policy by directly maximizing an objective function using gradient descent methods, in which the objective function is usually the expectation of accumulative future rewards. The latter indirectly finds the optimal policy by solving the Bellman equation, which is the sufficient and necessary condition from Bellman's principle of optimality. We study policy gradient forms of direct and indirect RL and show that both of them can derive the actor-critic architecture and can be unified into a policy gradient with the approximate value function and the stationary state distribution, revealing the equivalence of direct and indirect RL. We employ a Gridworld task to verify the influence of different forms of policy gradient, suggesting their differences and relationships experimentally. Finally, we classify current mainstream RL algorithms using the direct and indirect taxonomy, together with other ones including value-based and policy-based, model-based and model-free.

The investigation of queueing systems with impatient customers has a long history in queueing theory, with the now standard Erlang-A model going back at least to Palm’s classic 1957 paper. Despite this long record of work, focus on systems with abandonments has been rekindled due to recent interest in analyzing complex service systems, most notably large call centers. Interestingly, this new surge of work has been motivated both by applied and theoretical advances. In the applied realm, very large call centers are now an important component in many industrial sectors and advanced call management software enables the use of sophisticated algorithms in operating these centers. On the more theoretical side, there have been great advances in approximating large queueing systems with abandonments via many-server asymptotics and related diffusion approximations. Furthermore, there have been numerous advances in the understanding of flexible server models with abandonments, which serve as models for systems with “cross-trained” agents. Techniques from revenue management, game theory, and stochastic optimization have also been applied to the operation of such systems. In addition, queueing systems with impatient customers also have a rich connection with perishable inventory and organ transplantation systems. Thus the requirements of practitioners, and the myriad new tools of theoreticians, have motivated this focused issue on stochastic systems with abandonments. This special issue actually consists of nine papers, eight of which appear together here. Due to a publishing error, one of the papers, by Baris Ata and Mustafa H.

Ventricular assist device (VAD) implantation for patients with advanced heart failure as an alternative therapy to heart transplantation has increased patients' survival and quality of life thanks to improvements in the pump's geometry, local and supervisory control systems. Suppose the supervisory control system for these devices is considered an open system, in that case, an approach considers integrating various agents that interact collaboratively to determine the behavior. In addition to the physical device and its local control system, there is the patient and the medical team assisting this patient. In this way, a VAD can be considered a cyber-physical device, in which characteristics such as intelligence, connectivity, and reactivity can be observed. Based on this paradigm, there was a possibility of improvement from the interaction and collaboration between cyber-physical VADs capable of communicating with a supervisory control system. While the local control system is responsible for the device's drive functions, the supervisory system is responsible for managing several local control systems of each VAD. The connectivity between the devices is a fundamental characteristic for the proposal of intelligent VADs, which can offer the multidisciplinary medical team tools to monitor the patient's condition and decision support algorithms that make up the supervisory system. This new context causes an evolution from cyber-physical VADs to an architecture based on Reference Architecture Model for Industry 4.0 (RAMI 4.0) to allow the reconfiguration of VADs in a new paradigm based on Health 4.0.

Synthesis of optimal controllers for a class of maximization problems

Optimal control policies for reducing the maximum size of a closed epidemic—II. Stochastic dynamics

Development of the Schrödinger equation and Klein-Gordon equation via Bellman's principle of optimality

An uplink cognitive radio system with a single primary user (PU) and multiple secondary users (SUs) is considered. The SUs have an individual average delay constraint and an aggregate average interference constraint to the PU. If the interference channels between the SUs and the PU are statistically different due to the different physical locations of the SUs, the SUs will experience different delay performances. This is because SUs located closer to the PU transmit with lower power levels. A dynamic scheduling-and- power-allocation policy that uses the dynamic programming, is proposed. The proposed policy can provide the required average delay guarantees to all SUs irrespective of their location as well as protect the PU from harmful interference. The policy is shown to be asymptotically delay optimal in the light traffic regime. Motivated, by the high complexity of the dynamic programming algorithm in the optimal policy we exploit the structure of the problem's solution to present an alternative suboptimal policy. Through simulations we show that in both light and heavy traffic regimes, the delay performance of the suboptimal policy is within $0.3\%$ from the optimal policy and both outperforming existing methods.

This paper considers quickest detection scheme where the change in an underlying parameter influencing human decisions is to be detected by only observing the human decisions. Stemming from behavioral economics and mathematical psychology, we propose two generative models for the human decision maker. Namely, we consider an anticipatory decision making model and a quantum decision model. From a decision theoretic point of view, anticipatory models are time inconsistent, meaning that Bellman's principle of optimality does not hold. The appropriate formalism is thus the subgame Nash equilibrium. We show that the interaction between anticipatory agents and sequential quickest detection results in unusual (nonconvex) structure of the quickest change detection policy. In contrast the quantum decision model, despite its mathematical complexity, results in the typical convex quickest detection policy. The optimal quickest detection policy is shown to perform strictly worse than classical quickest detection for both models, via a Blackwell dominance argument. The model and structural results provided contribute to an understanding of the dynamics of human-sensor interfacing.

To estimate the number of stages contained in the planning horizon, while performing a decision task, a modified multistage betting game is considered. The version introduces a maximum final pay-off. Under the assumption of a logarithmic utility-function the optimal policy is derived. According to the number of anticipated stages, different suboptimal policies may be distinguished. In two experiments, a total of 100 Ss were investigated under four experimental conditions. The medians for the estimates of the length of the planning horizon are around two stages. The efficiencies of the according suboptimal policies were determined in a sensitivity analysis and were found to be approximately 90 percent.

The economic aspects of the online, programmable computer for supervisory control of producing operations are discussed. The material presented reflects the performance and economics of a pilot supervisory control installation that has been supervising and controlling producing operations for one year in two widely separated fields in southeastern New Mexico. Expansion of this pilot supervisory system is necessary to obtain an actual payout, but the performance of the pilot project does provide a reliable basis for predicting the economics of full scale computer controlled producing operations. Introduction A pilot installation to evaluate a supervisory, monitoring and control system for properties operated in southeastern New Mexico by Continental Oil Co. was installed in Oct., 1964. The pilot project included 56 wells in two fields to be controlled from the district office in Hobbs, N. M. Results of the pilot project would serve to evaluate the economics of expanding the system to include three additional areas and 326 additional wells in a full scale operation, with possible future expansion to a total of 600 to 700 wells. The pilot installation and three additional areas to be included in the full scale project are shown in Fig. 1.This supervisory control system is similar in concept to an earlier system installed during 1962 in the Big Horn Basin of Wyoming by the Rocky Mountain Region of Conoco. The pilot project has been sufficiently complex to prove equipment capability and performance, but limited enough to keep installation costs within reason should results indicate the total project to be economically unattractive leases in the Hobbs and El Mar fields of New Mexico comprise this two-field pilot project. These areas are separated by approximately 56 air miles (Fig. 1). The Hobbs field, located two miles from the central supervisory in the city of Hobbs, consists of three leases with 14 flowing or pumping wells producing 5 1 8 BOPD into one central tank battery. The El Mar field includes three leases with 42 pumping or flowing wells producing 700 BOPD into a central tank battery. The objective of the pilot program was to determine if a significant portion of the pumper functions could be performed by the supervisory system while maintaining producing efficiency at a level equal to or greater than before. It was recognized that not all of the pumper duties could be performed by computer control but that almost all of the check-and-see type routine could be surveyed by the system. The majority of pumper time involves travel and inspection duties; therefore, it was estimated that approximately 50 per cent of his assignments could be supervised by the control system. JPT P. 48ˆ

SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.

Supply or demand? Policy makers’ confusion in the presence of hysteresis

The penetration of renewable energy has been increasing rapidly through out the world. But its generation is intermittent unlike conventional sources. Energy storages provide an environmentally sustainable mechanism to reduce its variability. And given the geographically distributed nature of renewable energy sources, using a network of energy storages instead of using centralized ones can be helpful in avoiding congestion in transmission networks. In this paper we study a power system with such a network of storages. We provide a stochastic formulation to obtain the optimal scheduling policy for it. We show that the problem of calculating that policy is not convex in general and we find sufficient conditions for convexity. We also propose and numerically evaluate a heuristic algorithm which provides sub-optimal policies when the problem is non-convex.