Optimal transmission policies for two-user multiple access broadcast using dynamic team theory

Optimal consumption and portfolio policies when asset prices follow a diffusion process

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

In stochastic control, information structure arguments have been crucial for stochastic analysis. Such an approach is often called static reduction in dynamic team theory (or decentralized stochastic control) and has been an effective method for establishing existence and approximation results for optimal policies. In this paper, we classify such static reductions into three categories: (i) those that are policy-independent (introduced by Witsenhausen in [17]), (ii) those that are policy-dependent (introduced by Ho and Chu [7], [8] for partially nested dynamic teams), and (iii) static measurement with control-sharing reduction (where the measurements become static although control actions are shared according to the partially nested information structure). For these reductions, while there exist bijection relationships between glob-ally optimal solutions of dynamic teams and their reductions, in general there is no bijection for person-by-person optimal policies. We also establish a similar result (but not identical) concerning stationary solutions. We present sufficient conditions under which bijection relationships hold. Under static measurement with control-sharing reduction, connections between optimality concepts can be established under relaxed conditions. An implication is a convexity characterization of dynamic teams under static measurement with control-sharing reduction. Some counterparts for stochastic games are also discussed.

Motivated by the urgent demand for the electric vehicle (EV) fast refueling technologies, battery swapping and charging stations (BSCSs) are envisioned as a promising solution to provide timely EV refueling services. However, inappropriate battery charging operation in BSCSs cannot only incur unnecessary high charging cost but also threaten the reliability of the power grid. In this paper, we aim at obtaining an optimal charging operation policy for a single BSCS to minimize its charging cost while ensuring its quality-of-service. Leveraging the novel queueing network model, we propose to formulate the charging operation problem as a constrained Markov decision process and derive the optimal policy by the standard Lagrangian method and dynamic programming. To avoid the curse of dimensionality in practical large-scale systems, we further analyze the structure of the optimal policy and transform the dynamic programming procedure into an equivalent threshold optimization problem with a discrete separable convex objective function. Numerical results validate our theoretical analysis and the computational efficiency of our proposed algorithms. This paper also shows the impact of the system parameters (e.g., numbers of batteries and chargers) on the average cost under the optimal charging policy, which gives rich insights into the infrastructure planning of future BSCS networks.

A state-space process-control model involving adjustment error and deterministic drift of the process mean is presented. The optimal adjustment policy is developed by dynamic programming. This policy calls for a particular adjustment when a Kalman-filter estimator is outside a deadband defined by upper and lower action limits. The effects of adjustment cost, adjustment variance, and drift rate on the optimal policy are discussed. The optimal adjustment policy is computed for a real machining process, and a simulation study is presented that compares the optimal policy to two sensible suboptimal policies.

Determining the optimum replacement policy for Holstein dairy herds in Iran

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

We formulate and analyze a Markov decision process (dynamic programming) model for airline seat allocation (yield management) on a single-leg flight with multiple fare classes. Unlike previous models, we allow cancellation, no-shows, and overbooking. Additionally, we make no assumptions on the arrival patterns for the various fare classes. Our model is also applicable to other problems of revenue management with perishable commodities, such as arise in the hotel and cruise industries. We show how to solve the problem exactly using dynamic programming. Under realistic conditions, we demonstrate that an optimal booking policy is characterized by state- and time-dependent booking limits for each fare class. Our approach exploits the equivalence to a problem in the optimal control of admission to a queueing system, which has been well studied in the queueing-control literature. Techniques for efficient implementation of the optimal policy and numerical examples are also given. In contrast to previous models, we show that 1) the booking limits need not be monotonic in the time remaining until departure; 2) it may be optimal to accept a lower-fare class and simultaneously reject a higher-fare class because of differing cancellation refunds, so that the optimal booking limits may not always be nested according to fare class; and 3) with the possibility of cancellations, an optimal policy depends on both the total capacity and the capacity remaining. Our numerical examples show that revenue gains of up to 9% are possible with our model, compared with an equivalent model omitting the effects of cancellations and no-shows. We also demonstrate the computational feasibility of our approach using data from a real-life airline application.

We consider a physical asset consisting of complex systems, where the systems may require upgrades during the lifetime of the asset. In practice, the asset owner and system supplier can make the upgrade decisions together, requiring a decision-support model that can be jointly used to optimize the total benefit for both parties. Motivated by a real-life use case including an asset owner and a system supplier, we build a continuous-time model to optimize the upgrade decisions of a system during the fixed lifetime of the asset. In our model, we capture the key critical factors that drive the upgrade decisions: increasing functionality requirements due to evolving technology, age-dependent maintenance costs, a predetermined overhaul plan of the asset, and the lifetime of the asset. A system upgrade is less costly if it is executed jointly with an asset overhaul. We first analyze the case with no additional cost of upgrading outside an overhaul. We analytically characterize the structure of the optimal upgrade policy under various realistic assumptions that lead to different types of cost functions. We then use these results as a building block to characterize the optimal policy for a generalized cost function. When there is a penalty for upgrading outside an overhaul moment, we propose a dynamic programming approach that efficiently determines the optimal upgrade policy by using our analytical results. We also prove that as this penalty increases, the optimal policy can only change to one where the number of upgrades not jointly executed with overhauls is reduced. However, the optimal number of upgrades is a nonincreasing function of this penalty. Also, surprisingly, more overhauls may lead to a smaller number of upgrades under the optimal policy. Funding: This publication is part of the project “Maritime Remote Control Tower for Service Logistics Innovation (MARCONI)” (project 439.18.309) of the research program “Integrator-Logistics as Enabler for Enhancing Society,” which is (partly) financed by the Dutch Research Council (NWO).

In this paper, two new power policies (PPs), namely the Rate Maximization Policy (RMP) and the Rate-Bound Maximization Policy (RBMP), for Cognitive Radio (CR) systems are developed based on an expected rate maximization criterion. Both policies are designed such as to satisfy a constraint related to the average interference caused by the CR network to primary communication. The key novelty here is that the proposed PPs take into account the existence of limited (hybrid) channel feedback, where the direct secondary channel is known instantaneously while the interference caused by the primary transmission to the secondary receiver is only known statistically. The optimal policy (RMP) is characterized, and a low complexity algorithm is presented that allows for its efficient and accurate implementation. The two policies are compared and RMP is shown to lead to substantial energy consumption savings at equal rate performance.

One most common problem in reinforcement learning systems (e.g. Q-learning) is to reduce the number of trials to converge to an optimal policy. As one of the solution to the problem, k-certainty exploration method was proposed. Miyazaki reported that this method could determine an optimal policy faster than Q-learning in Markov decision processes (MDPs). This method is very efficient learning method. But, we propose an improvement plan that makes this method more efficient. In k-certainty exploration method, in case there is no k-uncertainty rule (rules which aren’t selected more than k times) in current state, an agent sometimes walks randomly until it finds a state where it can select a k-uncertainty rule. We think that this behavior is not efficient. To reduce this useless behavior, we propose combining k-certainty exploration method with Dynamic programming (DP). Miyazaki’s system uses DP after all rules are executed at least k times. But, our method uses k-certainty exploration method along with DP during learning. Our method, takes two pattern actions. In case an agent can select k-uncertainty rules, one of these rules is selected at random as k-certainty exploration method. In another case there is no k-uncertainty rule, behavior of agent is different from the behavior of k-certainty exploration method. In that case, our method uses DP to compute an optimal policy for moving from a current state to a state in which some k-uncertainty rules remain. The model for DP is constructed by using only known states. The outline will be described below. First, an agent makes a map constructed by using only known states. In the map, goals are states in which there are k-uncertainty rules and arbitrary state values are set in these states. Point to which attention should be paid is that the map is not given from outside. It is made from only experience. Next, each state values of states in which there are only k-certainty rules in the map are computed by DP (we used Policy Iteration). Finally, an agent continues to select greedy action until it arrives at a state in which it can select a k-uncertainty rule. By this improvement, we expect it can determine an optimal policy faster than k-certainty exploration method. And we have verified that our exploration method can determine an optimal policy faster than k-certainty exploration method by computer simulation.

Optimal production-inventory policy for the multi-period fixed proportions co-production system

The cost-effectiveness and value of additional information about a health technology or program may change over time because of trends affecting patient cohorts and/or the intervention. Delaying information collection even for parameters that do not change over time may be optimal. We present a stochastic dynamic programming approach to simultaneously identify the optimal intervention and information collection policies. We use our framework to evaluate birth cohort hepatitis C virus (HCV) screening. We focus on how the presence of a time-varying parameter (HCV prevalence) affects the optimal information collection policy for a parameter assumed constant across birth cohorts: liver fibrosis stage distribution for screen-detected diagnosis at age 50. We prove that it may be optimal to delay information collection until a time when the information more immediately affects decision making. For the example of HCV screening, given initial beliefs, the optimal policy (at 2010) was to continue screening and collect information about the distribution of liver fibrosis at screen-detected diagnosis in 12 years, increasing the expected incremental net monetary benefit (INMB) by $169.5 million compared to current guidelines. The option to delay information collection until the information is sufficiently likely to influence decisions can increase efficiency. A dynamic programming framework enables an assessment of the marginal value of information and determines the optimal policy, including when and how much information to collect.

Bernoulli multi-armed bandits are a reinforcement learning model used to study a variety of choice optimization problems. Often such optimizations concern a finite-time horizon. In principle, statistically optimal policies can be computed via dynamic programming, but doing so is considered infeasible due to prohibitive computational requirements and implementation complexity. Hence, suboptimal algorithms are applied in practice, despite their unknown level of suboptimality. In this article, we demonstrate that optimal policies can be efficiently computed for large time horizons or number of arms thanks to a novel memory organization and indexing scheme. We use optimal policies to gauge the suboptimality of several well-known finite- and infinite-time horizon algorithms including Whittle and Gittins indices, epsilon-greedy, Thompson sampling, and upper-confidence bound (UCB) algorithms. Our simulation study shows that all but one evaluated algorithm perform significantly worse than the optimal policy. The Whittle index offers a nearly optimal strategy for multi-armed Bernoulli bandits despite its suboptimal decisions—up to 10%—compared to an optimal policy table. Lastly, we discuss optimizations of known algorithms. We derive a novel solution from UCB1-tuned. It outperforms other infinite-time horizon algorithms when dealing with many arms. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>Impact statement</i>—Bernoulli bandits are a reinforcement learning model used to improve decisions with binary outcomes. They have various applications ranging from headline news selection to clinical trials. Existing bandit algorithms are suboptimal. This article provides the first practical computation method, which determines the optimal decisions in Bernoulli bandits. It provides the lowest achievable decision regret (maximum expected benefit). In clinical trials, where an algorithm selects treatments for subsequent patients, our method can substantially reduce the number of unsuccessfully treated patients—by up to 5×. The optimal strategy is also used for new comprehensive evaluations of well-known suboptimal algorithms. This can significantly improve decision effectiveness in various applications.

One of the major challenges in Dynamic Spectrum Access (DSA) systems is to guarantee a required level of Quality of Service (QoS) to secondary users of the spectrum. In this paper, we propose efficient algorithms for deriving optimal policies for the sensing / transmitting trade-off in hardware-constrained DSA systems. Unlike previous approaches which seek to maximize mean data rate for the secondary users, the proposed algorithms derive policies which minimize the probability of excessive queuing delays. Large Deviations (LD) asymptotics are used to approximate the probability of interest and policies maximizing the associated LD exponent are proposed. Although dynamic programming is not able to identify the optimal policy in this case, much more efficient algorithms than exhaustive search are proposed. These algorithms are based on specific properties of the optimal policy which are described and proven in this paper.