Adaptive Policies in Markov Decision Processes with Uncertain Transition Matrices

This paper extends finite state and action space Markov Decision Process (MDP) models by introducing a new type of measurement for the outcomes of actions. The new measurement allows to sequentially observe the next-state transition for taking an action, i.e., the actions are ordered and the next action outcome in the sequence is observed only if the current action is not chosen. The sequentially-observed MDP (SO-MDP) shares some properties with a standard MDP: among history dependent policies, Markovian ones are still optimal. SO-MDP policies have the advantage of producing better rewards than standard optimal MDP policies due to additional measurements. Computing these policies, on the other hand, is more complex and we present a linear programming based synthesis of the optimal decision policies for the finite horizon SO-MDPs. A simulation example of multiple autonomous agents is also provided to demonstrate the SO-MDP model and the proposed policy synthesis method.

We propose a power and constellation-size adaptation scheme for the multicast system employing multilevel quadrature amplitude modulation (MQAM) with discrete constellation sizes, which aims at achieving high multicast goodput over wireless fading channels. Specifically, we formulate the optimization problem to maximize multicast goodput via adaptive transmission-mode selection, including power and constellation-size adaptation, and then solve for the optimal adaptation policy through two steps. We first investigate the goodput variation with the transmission-mode selection and equivalently convert the optimization problem into a concave-like form by introducing the concepts of effective marginal goodput gain (EMGG) and adjacent EMGG. Then, we derive the optimal transmission adaptation policy by using a global adjacent EMGG threshold. Finally, simulation results are provided to show the significant multicast goodput improvement gained by applying the optimal power and constellation-size adaption.

The main objective of this paper is to synthesize optimal decision-making policies for a finite-horizon Markov Decision Process (MDP) while satisfying a safety constraint that imposes an upper bound on the state probability density function (pdf) of the underlying Markov Chain (MC) for all time steps. The classical approach based on state-action frequencies for constrained MDPs yields decision policies that provide safety constraint satisfaction for stationary distributions (i.e., asymptotically), but not necessarily providing safety during the transient regime. This paper introduces a new synthesis method for randomized Markovian policies for finite-horizon MDPs, where the safety constraint satisfaction is guaranteed for both the transient and the stationary distributions independent from initial state (i.e., providing safe policies for the worst-case analysis). An efficient Linear Programming (LP) based synthesis algorithm is proposed, which produces a convex set of feasible policies and ensures that the expected total reward is above a computable lower-bound. A simulation example of a swarm of autonomous agents is also presented to demonstrate the practical importance of having safe policies.

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.

Many of simulation based learning algorithms have been developed to obtain near optimal policies for Markov decision processes (MDPs) with large state space. However, most of them are for unichain problems. In view that some applications involve multichain processes and it is NP-hard to determine whether a MDP is unichain or not, it is desirable to obtain an algorithm that is applicable to multichain problems as well. This paper presents a rollout algorithm for multichain MDPs with average cost. Preliminary analysis of the estimation error and parameter settings are provided based on the problem structures, i.e., mixing time of transition matrix. Ordinal optimization and Optimal Computing Budget Allocation are also suggested to improve the efficiency of the algorithm.

We study stochastic variants of Packing Integer Programs (PIP) --- the problems of finding a maximum-value 0/1 vector x satisfying Ax ≤ b, with A and b nonnegative. Many combinatorial problems belong to this broad class, including the knapsack problem, maximum clique, stable set, matching, hypergraph matching (a.k.a. set packing), b-matching, and others. PIP can also be seen as a multidimensional knapsack problem where we wish to pack a maximum-value collection of items with vector-valued sizes. In our stochastic setting, the vector-valued sizes of each item is known to us apriori only as a probability distribution, and the size of an item is instantiated once we commit to including the item in our solution.Following the framework of [3], we consider both adaptive and non-adaptive policies for solving such problems, adaptive policies having the flexibility of being able to make decisions based on the instantiated sizes of items already included in the solution. We investigate the adaptivity gap for these problems: the maximum ratio between the expected values achieved by optimal adaptive and non-adaptive policies. We show tight bounds on the adaptivity gap for set packing and b-matching, and we also show how to find efficiently non-adaptive policies approximating the adaptive optimum. For instance, we can approximate the adaptive optimum for stochastic set packing to within O(d1/2), which is not only optimal with respect to the adaptivity gap, but it is also the best known approximation factor in the deterministic case. It is known that there is no polynomial-time d1/2-e approximation for set packing, unless NP = ZPP. Similarly, for b-matching, we obtain algorithmically a tight bound on the adaptivity gap of O(λ) where λ satisfies Σ λbj+1 = 1.For general Stochastic Packing, we prove that a simple greedy algorithm provides an O(d)-approximation to the adaptive optimum. For A ∈ [0, 1]dxn, we provide an O(λ) approximation where Σ 1/λbj = 1. (For b = (B, B,..., B), we get λ = d1/B.) We also improve the hardness results for deterministic PIP: in the general case, we prove that a polynomial-time d1-e-approximation algorithm would imply NP = ZPP. In the special case when A ∈ [0,1]dxn and b = (B,B,...,B), we show that a d1/B-∈-approximation would imply NP = ZPP. Finally, we prove that it is PSPACE-hard to find the optimal adaptive policy for Stochastic Packing in any fixed dimension d ≥ 2.

An Operational Framework for the Adoption and Integration of New Diagnostic Tests into Emergency Department Workflow

The maximum possible throughput (rate of task completion) of a multi-server system is typically the sum of the service rates of individual servers. Recent works show that task replication can boost the throughput, in particular if the service time has high variability (Cv > 1). Thus, redundancy can be used to create synergy among servers such that their overall throughput is greater than sum of individual servers. This paper seeks to find the fundamental limit of this capacity boost achieved by task replication. The optimal adaptive replication policy can be found using a Markov Decision Process (MDP) framework, but the MDP is hard to solve in general. We propose two replication policies, MaxRate and AdaRep that gradually add replicas only when needed. To quantify the optimality gap of these policies, we also derive an a upper bound on the service capacity for the two-server case.

Computing monotone policies for Markov decision processes: a nearly-isotonic penalty approach

Q-learning is a very popular reinforcement learning algorithm being proven to converge to optimal policies in Markov decision processes. However, Q-learning shows artifacts if the optimal action is played with a low probability, a situation that may arise due to wrong initialization of Q-values or due to convergence to an almost pure policy after which a change in the environment makes another action optimal. These artifacts was resolved in literature by the variant Frequency Adjusted Q-learning (FAQL). However, FAQL also suffered from practical concerns that limited the policy subspace for which the behavior was improved. Here, we introduce the Repeated Update Q-learning (RUQL), a variant of Q-learning that resolves the undesirable artifacts of Q-learning without the practical concerns of FAQL. We show (both theoretically and experimentally) the similarities and differences between RUQL and FAQL (the closest state-of-the-art). Experimental results verify the theoretical insights and show how RUQL outperforms FAQL and QL in non-stationary environments.

In this research work, the authors derive closed‐form expressions for capacity per unit bandwidth (spectrum efficiency) of correlated Rayleigh fading channels under maximal ratio combining diversity at low signal‐to‐noise ratios (SNRs). Spectrum efficiency expressions are derived for M diversity branches under adaptation policies, such as: (i) optimal power and rate adaptation (OPRA) policy, (ii) optimal rate adaptation (ORA) policy and (iii) truncated channel inversion with fixed rate (TIFR) policy. Even at low SNRs, if space diversity is exercised using a single‐input multiple‐output system, spectrum efficiency achieved is higher compared with that achieved when signals are uncorrelated and have no diversity. This forms the focal point of this study. Analytical results show accurately that OPRA policy provides highest capacity over other adaptation policies. The spectrum efficiency for the three policies and outage probability for the low SNR case are derived, plotted and analysed in detail in this work.

Economic MPC of Markov Decision Processes: Dissipativity in undiscounted infinite-horizon optimal control

The maximum possible throughput (or the rate of job completion) of a multi-server system is typically the sum of the service rates of individual servers. Recent work shows that launching multiple replicas of a job and canceling them as soon as one copy finishes can boost the throughput, especially when the service time distribution has high variability. This means that redundancy can, in fact, create synergy among servers such that their overall throughput is greater than the sum of individual servers. This work seeks to find the fundamental limit of the throughput boost achieved by job replication and the optimal replication policy to achieve it. While most previous works consider upfront replication policies, we expand the set of possible policies to delayed launch of replicas. The search for the optimal adaptive replication policy can be formulated as a Markov Decision Process, using which we propose two myopic replication policies, MaxRate and AdaRep, to adaptively replicate jobs. In order to quantify the optimality gap of these and other policies, we derive upper bounds on the service capacity, which provide fundamental limits on the throughput of queueing systems with redundancy.

We consider the problem of synthesizing optimal policies for Markov decision processes (MDP) for both utility objective and security constraint. Specifically, our goal is to maximize the entropy rate of the MDP while achieving a surveillance task in the sense that a given region of interest is visited infinitely often with probability one (w.p.1). Such a policy is of our interest since it guarantees both the completion of tasks and maximizes the unpredictability of limit behavior of the system. Existing works either focus on the total entropy or do not consider the surveillance tasks which are not suitable for surveillance tasks with infinite horizon. We provide a complete solution to this problem. Specifically, we present an algorithm for synthesizing entropy rate maximizing policies for communicating MDPs. Then based on a new state classification method, we show the entropy rate maximization problem under surveillance task can be effectively solved in polynomial-time. We illustrate the proposed algorithm based on a case study of robot planning scenario.

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation