A Gradient-Aware Search Algorithm for Constrained Markov Decision Processes.

This letter investigates an unmanned aerial vehicle (UAV)-aided data collection system, where a UAV is deployed to gather information from terrestrial Internet of Things (IoT) devices. We aim to minimize the mission completion time by optimizing the UAV trajectory, while ensuring all the target data can be successfully collected with a given energy budget. Firstly, the problem is formulated as a constrained Markov Decision Process (CMDP). Then, a constrained soft actor-critic (CSAC) algorithm is proposed by incorporating Lagrangian primal-dual optimization (PDO) into the soft actor-critic (SAC) framework. Simulation results demonstrate that the proposed algorithm outperforms state-of-the-art benchmark algorithms in terms of the mission completion time. Particularly, it is able to learn an adaptive policy that outputs optimal trajectories for different device locations.

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

Several algorithms already exist for solving the uncapacitated facility location problem. The most efficient are based upon the solution of the strong linear programming relaxation. The dual of this relaxation has a condensed form which consists of minimizing a certain piecewise linear convex function. This paper presents a new method for solving the uncapacitated facility location problem based upon the exact solution of the condensed dual via orthogonal projections. The amount of work per iteration is of the same order as that of a simplex iteration for a linear program inm variables and constraints, wherem is the number of clients. For comparison, the underlying linear programming dual hasmn + m + n variables andmn +n constraints, wheren is the number of potential locations for the facilities. The method is flexible as it can handle side constraints. In particular, when there is a duality gap, the linear programming formulation can be strengthened by adding cuts. Numerical results for some classical test problems are included.

Dynamic principal agent models are formulated based on constrained Markov decision process (CMDP), in which conditions are given that the state space of the system is countable and the agent chooses his actions from a countable action set. If the principal has finite alternative contracts to select, it is shown that the optimal contract solution and the corresponding optimal policy can be obtained by linear programming under the discounted criterion and average criterion.

This paper is motivated by the requirement of high information freshness in the industrial Internet of Things (IIoT). An industrial robot sends short status packets to a control center (CC), and the timeliness of status updates is measured by the age of information (AoI). Due to the dynamic change of the wireless channel, the robot needs to send a pilot for channel estimation during each coherence time. Considering the robot is energy-limited, we investigate the average AoI minimization scheme for short packet communications under the average power consumption constraint. By rationally analyzing state transitions, we first formulate the problem as a constrained Markov Decision Process, and obtain the optimal solution through linear programming (LP). Then, for the problem of high computational complexity caused by too many variables in LP, we propose a heuristic threshold-based status update scheme by exploiting the threshold structure of the optimal solution. Simulation results show that the LP scheme can effectively minimize the average AoI and the threshold-based scheme can achieve near-optimal performance. Interestingly, we find that when the channel suffers from severe fading, at the end of a coherence time, the robot does not send status packets even if the information at the CC is particularly stale.

We present a novel linear programming (LP) based approach for efficiently solving the intensity modulated radiation therapy (IMRT) fluence-map optimization (FMO) problem to global optimality. Our model overcomes the apparent limitations of a linear-programming approach by approximating any convex objective function by a piecewise linear convex function. This approach allows us to retain the flexibility offered by general convex objective functions, while allowing us to formulate the FMO problem as a LP problem. In addition, a novel type of partial-volume constraint that bounds the tail averages of the differential dose–volume histograms of structures is imposed while retaining linearity as an alternative approach to improve dose homogeneity in the target volumes, and to attempt to spare as many critical structures as possible. The goal of this work is to develop a very rapid global optimization approach that finds high quality dose distributions. Implementation of this model has demonstrated excellent results. We found globally optimal solutions for eight 7-beam head-and-neck cases in less than 3 min of computational time on a single processor personal computer without the use of partial-volume constraints. Adding such constraints increased the running times by a factor of 2–3, but improved the sparing of critical structures. All cases demonstrated excellent target coverage (>95%), target homogeneity (<10% overdosing and <7% underdosing) and organ sparing using at least one of the two models.

Sleeping experts and bandits approach to constrained Markov decision processes

The searching denotes a process that aims to find a specific item in the data set. After performing the process of searching, the desired items are likely found or not. This kind of investigations was actually demonstrated by some researchers. For each search algorithm, the previous studies just focused to obtain the search time but did not obtain the number of passed-nodes. Furthermore, the results obtained with regards to the search time were not reported systematically. Hence, this study investigates how fast the desired items were found and how few passed-nodes were obtained when searching desired items by using three search algorithms. These algorithms were linear, binary, and interpolation algorithms. By using search algorithms, one hundred twenty desired items were searched in a collection of active-student data (3946 records in total). The search time needed to find a specific item and the number of passed nodes were recorded. The results showed that the searching process with linear search algorithm required more time and more passed-nodes than binary and interpolation search algorithms. Following this, searching time and the number of passed-nodes for desired items were significantly smaller for binary search than for the interpolation search algorithm. Taken together, the present results suggested that the binary search could be considered as a solid algorithm in searching a specific item in the data set utilized by the current study.

In this paper, a power saving scheme is investigated for macrocell-assisted small cell networks in the downlink, in which small cells are selected to stay in sleep or wake-up modes in order to mitigate excessive power consumption and the small cells operating in the sleep mode are eligible to off-load their served users to the macrocell to preserve communications. By taking the dynamics of user arrival into account, we attempt to minimize the average total power consumption of the network, which depends on several network parameters like power consumption of base stations, available bandwidth, user load in the cells, cell size, user rate requirement, rate outage probability and noise power density, while ensuring the average number of dropping off-loaded users to a satisfactory level. The design problem is then formulated as a constrained Markov decision process and solved via linear programming. A randomized strategy is proposed to accomplish the optimal sleep/wake-up policy for small cells. Simulation results confirm the effectiveness of the proposed scheme, as compared with the one without adopting sleep/wake-up mechanisms, and help us to capture the impact of the network parameters on the entire power saving.

Delay optimal for reliability-guaranteed concurrent transmissions with raptor code in multi-access 6G edge network

The problem of dynamically accessing a set of parallel channels occupied by primary users is considered. The secondary user is allowed to sense and to transmit in a single channel. By exploiting idle periods between bursty transmissions of primary users, and by using a periodic sensing strategy, optimal dynamic access is achieved by maximizing the throughput of the secondary user while constraining collision probability with the primary user. The optimal dynamic spectrum access problem can then be formulated within the framework of constrained Markov decision processes (CMDPs). The optimal control policy is identified via a linear program, and its performance is analyzed numerically and through Monte Carlo simulations. Finally, we compare the optimal scheme to an ideal benchmark case when simultaneous sensing of all channels is assumed.

To balance the tradeoff between quality of service (QoS) and operating expenditure (OPEX), the service provider should request the appropriate amount of resources to the cloud operator based on the estimated variation of service requests. This paper proposes a popularity-aware service provisioning framework (PASPF), which leverages the network data analytics function (NWDAF) to obtain analytics on service popularity variations. These analytics estimate the congestion level and list of top services contributing most of the traffic change. Based on the analytics, PASPF enables the service provider to request the appropriate amount of resources for each service for the next time period to the cloud operator. To minimize the OPEX of the service provider while keeping the average response time of the services below their requirements, we formulate a constrained Markov decision process (CMDP) problem. The optimal stochastic policy can be obtained by converting the CMDP model into a linear programming (LP) model. Evaluation results demonstrate that the PASPF can achieve less than 50% OPEX of the service provider compared to a popularity-non-aware scheme while keeping the average response time of the services below the requirement.

Solving (mixed) integer (linear) programs, (M)I(L)Ps for short, is a fundamental optimisation task with a wide range of applications in artificial intelligence and computer science in general. While hard in general, recent years have brought about vast progress for solving structurally restricted, (non-mixed) ILPs: n-fold, tree-fold, 2-stage stochastic and multi-stage stochastic programs admit efficient algorithms, and all of these special cases are subsumed by the class of ILPs of small treedepth. In this paper, we extend this line of work to the mixed case, by showing an algorithm solving MILP in time f(a,d)poly(n), where a is the largest coefficient of the constraint matrix, d is its treedepth, and n is the number of variables. This is enabled by proving bounds on the denominators (fractionality) of the vertices of bounded-treedepth (non-integer) linear programs. We do so by carefully analysing the inverses of invertible sub-matrices of the constraint matrix. This allows us to afford scaling up the mixed program to the integer grid, and applying the known methods for integer programs. We then trace the limiting boundary of our "bounded fractionality" approach both in terms of going beyond MILP (by allowing non-linear objectives) as well as its usefulness for generalising other important known tractable classes of ILP. On the positive side, we show that our result can be generalised from MILP to MIP with piece-wise linear separable convex objectives with integer breakpoints. On the negative side, we show that going even slightly beyond such objectives or considering other natural related tractable classes of ILP leads to unbounded fractionality. Finally, we show that restricting the structure of only the integral variables in the constraint matrix does not yield tractable special cases.

Thanks to smart manufacturing and artificial intelligence technologies, unmanned aerial vehicles (UAVs) are envisioned to play a critical role in future Internet of things (IoT) networks to execute data collection tasks. In this article, we leverage age of information (AoI) to measure the freshness of data packets received by the UAV from IoT sensors. Considering the heterogeneity of IoT devices, we aim to minimize the weighted sum AoI by jointly optimizing the UAV's trajectory and IoT devices association in UAV-assisted IoT networks, where the UAV's cumulative propulsion energy cost is limited by the battery capacity. Since the optimization object is confined by a set of short-term constraints and a long-term constraint, this problem is modeled as a constrained Markov decision process (CMDP). We leverage safe actor-critic (Safe-AC) to solve the CMDP. To satisfy the mixed constraints, the safe policy set of Safe-AC is induced by a Lyapunov function, thereafter, a policy distillation technology is leveraged to search the optimal policy. Experimental results indicate that our proposed scheme can strictly satisfy the propulsion energy cost budget requirement at the expense of around 2% loss of the reward compared to baseline methods.

The theme of this paper is the application of linear analysis to simplify and extend convex analysis. The central problem treated is the standard convex program — minimize a convex function subject to inequality constraints on other convex functions. The present approach uses the support planes of the constraint region to transform the convex program into an equivalent linear program. Then the duality theory of infinite linear programming shows how to construct a new dual program of bilinear type. When this dual program is transformed back into the convex function formulation it concerns the minimax of an unconstrained Lagrange function. This result is somewhat similar to the Kuhn—Tucker theorem. However, no constraint qualifications are needed and yet perfect duality maintains between the primal and dual programs.