Deterministic policy gradient algorithms for semi‐Markov decision processes

In this work, we propose a Model Predictive Control (MPC)-based Reinforcement Learning (RL) method for Autonomous Surface Vehicles (ASVs). The objective is to find an optimal policy that minimizes the closed-loop performance of a simplified freight mission, including collision-free path following, autonomous docking, and a skillful transition between them. We use a parametrized MPC-scheme to approximate the optimal policy, which considers path-following/docking costs and states (position, velocity)/inputs (thruster force, angle) constraints. The Least Squares Temporal Difference (LSTD)-based Deterministic Policy Gradient (DPG) method is then applied to update the policy parameters. Our simulation results demonstrate that the proposed MPC-LSTD-based DPG method could improve the closed-loop performance during learning for the freight mission problem of ASV.

A unified approach to time-aggregated Markov decision processes

Optimality problems in infinite horizon discrete time vector criterion Markov and semi-Markov decision processes can be expressed as standard problems of multiple objective linear programming. “VMDP” is a FORTRAN program for enumerating the objective function coefficients and constraint coefficients in these problems and finding all nondominated extreme points, i.e. all nondominated pure stationary strategies, and all nondominated edges. Using VMDP, computational experiments are presented. The characteristics of Markov and semi-Markov decision problems (the VMDP code) with those of general muitiple objective linear programming problems (the Steuer Adbase code) are compared.

Reinforcement learning (RL) is a state or action value based machine learning method which approximately solves large-scale Markov Decision Process (MDP) or Semi-Markov Decision Process (SMDP). A multi-step RL algorithm called Sarsa(,k) is proposed, which is a compromised variation of Sarsa and Sarsa(). It is equivalent to Sarsa if k is 1 and is equivalent to Sarsa() if k is infinite. Sarsa(,k) adjust its performance by setting k value. Two forms of Sarsa(,k), forward view Sarsa(,k) and backward view Sarsa(,k), are constructed and proved equivalent in off-line updating.

We consider a human agent servicing a queue of homogeneous tasks. The agent can service a task with normal or high fidelity level, where fidelity refers to the degree of exactness and precision while servicing the task. We assume the parameters of the human's service time distribution depend on the selected fidelity level and her cognitive state and are assumed to be unknown a priori. These parameters are learned online through Bayesian parameter estimation. We formulate a robust adaptive semi-Markov decision process (SMDP) to solve our optimal fidelity selection problem and extend the results on convergence of robust-adaptive Markov decision processes (MDP) to robust-adaptive SMDPs.

Multi-access edge computing (MEC) enables mobile applications, which consists of multiple dependent subtasks, to be executed in full parallelism between edge servers and mobile devices. The idea is to offload some of the subtasks to edge servers while utilizing the directed acyclic graphs (DAGs) based subtask dependency. In some cases involving multiple users running the same application, users can cooperate to improve the application’s performance (e.g., object detection accuracy in connected and autonomous vehicles) by sharing the intermediate results of the subtasks, which is referred to as cooperation gain. However, inter-user cooperation also introduces additional dependencies between the subtasks of different users, which inevitably increases the execution delay of the application with DAG tasks offloading. Therefore, how to jointly optimize execution delay and cooperation gain remains an open question. In this paper, we present an approach for DAG tasks offloading in MEC with inter-user cooperation to optimize the execution delay of the application and the cooperation gain. First, we introduce the concept of application utility, which incorporates both delay and cooperation gain. We formulate the DAG tasks offloading problem with inter-user cooperation as a Markov decision process (MDP) to maximize the total application utility. Next, we propose a quantized soft actor-critic (QSAC) algorithm to solve the formulated problem. Specifically, QSAC generates multiple potential solutions according to the order-preserving quantizing method, increasing the exploration efficiency to avoid falling into local optimal while maintaining the overall stability. Finally, simulation results validate that the proposed algorithm significantly improves the total application utility compared with the other benchmark algorithms.KeywordsMulti-access edge computing (MEC)Tasks offloadingDirected acyclic graphs (DAGs)Inter-user cooperationMarkov decision process (MDP)Soft actor-critic (SAC)

SummaryOne desired aspect of microservice architecture is the ability to self‐adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self‐adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE‐K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q‐learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE‐K model offers distributed microservice architecture with self‐adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q‐learning and policy gradient algorithms, including (1) a deep Q‐learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services‐oriented architecture and offers better performance than other dynamic decision‐making algorithms.

In this article, we propose an architecture of a bio-inspired controller that addresses the problem of learning different locomotion gaits for different robot morphologies. The modeling objective is split into two: baseline motion modeling and dynamics adaptation. Baseline motion modeling aims to achieve fundamental functions of a certain type of locomotion and dynamics adaptation provides a “reshaping” function for adapting the baseline motion to desired motion. Based on this assumption, a three-layer architecture is developed using central pattern generators (CPGs, a bio-inspired locomotor center for the baseline motion) and dynamic motor primitives (DMPs, a model with universal “reshaping” functions). In this article, we use this architecture with the actor-critic algorithms for finding a good “reshaping” function. In order to demonstrate the learning power of the actor-critic based architecture, we tested it on two experiments: (1) learning to crawl on a humanoid and, (2) learning to gallop on a puppy robot. Two types of actor-critic algorithms (policy search and policy gradient) are compared in order to evaluate the advantages and disadvantages of different actor-critic based learning algorithms for different morphologies. Finally, based on the analysis of the experimental results, a generic view/architecture for locomotion learning is discussed in the conclusion.

Multi-objective optimization models that encode ordered sequential constraints provide a solution to model various challenging problems including encoding preferences, modeling a curriculum, and enforcing measures of safety. A recently developed theory of topological Markov decision processes (TMDPs) captures this range of problems for the case of discrete states and actions. In this work, we extend TMDPs towards continuous spaces and unknown transition dynamics by formulating, proving, and implementing the policy gradient theorem for TMDPs. This theoretical result enables the creation of TMDP learning algorithms that use function approximators, and can generalize existing deep reinforcement learning (DRL) approaches. Specifically, we present a new algorithm for a policy gradient in TMDPs by a simple extension of the proximal policy optimization (PPO) algorithm. We demonstrate this on a real-world multiple-objective navigation problem with an arbitrary ordering of objectives both in simulation and on a real robot.

As a representative of the new generation of information technology, reinforcement learning technology has received more and more attention, but no scholars have yet applied reinforcement learning recommendation technology to ideological and political education. In this paper, a diversity optimization recommendation algorithm for ideological and political education is constructed, and the vector representation model of the state is designed and optimized based on the Markov decision process. Through the actor-critic algorithm, to calculate the probability of recommendation sampling to generate a recommendation list and then establish a decision model in order to improve the ideological and political education recommendation algorithm on the user’s feedback of accurate judgement and recommendation decision-making effect. The experimental results show that the Diversity Opt performs well in terms of prediction error. The algorithm is ahead of the comparison algorithm by an average of 9.35% and 8.3%, respectively, and all other performances are also significantly better than the comparison algorithm. The average score of students’ performance in the experimental class applying the algorithm is 14.4 points higher than that of the control class. It shows that the recommendation algorithm helps to accurately recommend ideological and political education and exhibits beneficial practical effects.

A constrained optimization perspective on actor–critic algorithms and application to network routing

Optimal causal policies maximizing the time-average reward over a semi-Markov decision process (SMDP), subject to a hard constraint on a time-average cost, are considered. Rewards and costs depend on the state and action, and contain running as well as switching components. It is supposed that the state space of the SMDP is finite, and the action space compact metric. The policy determines an action at each transition point of the SMDP. Under an accessibility hypothesis, several notions of time average are equivalent. A Lagrange multiplier formulation involving a dynamic programming equation is utilized to relate the constrained optimization to an unconstrained optimization parametrized by the multiplier. This approach leads to a proof for the existence of a semi-simple optimal constrained policy. That is, there is at most one state for which the action is randomized between two possibilities; at all other states, an action is uniquely chosen for each state. Affine forms for the rewards, costs and transition probabilities further reduce the optimal constrained policy to ‘almost bang-bang’ form, in which the optimal policy is not randomized, and is bang-bang except perhaps at one state. Under the same assumptions, one can alternatively find an optimal constrained policy that is strictly bang-bang, but may be randomized at one state. Application is made to flow control of a birth-and-death process (e.g., an M/M/s queue); under certain monotonicity restrictions on the reward and cost structure the preceding results apply, and in addition there is a simple acceptance region.

When animals have to make a number of decisions during a limited time interval, they face a fundamental problem: how much time they should spend on each decision in order to achieve the maximum possible total outcome. Deliberating more on one decision usually leads to more outcome but less time will remain for other decisions. In the framework of sequential sampling models, the question is how animals learn to set their decision threshold such that the total expected outcome achieved during a limited time is maximized. The aim of this paper is to provide a theoretical framework for answering this question. To this end, we consider an experimental design in which each trial can come from one of the several possible “conditions.” A condition specifies the difficulty of the trial, the reward, the penalty and so on. We show that to maximize the expected reward during a limited time, the subject should set a separate value of decision threshold for each condition. We propose a model of learning the optimal value of decision thresholds based on the theory of semi-Markov decision processes (SMDP). In our model, the experimental environment is modeled as an SMDP with each “condition” being a “state” and the value of decision thresholds being the “actions” taken in those states. The problem of finding the optimal decision thresholds then is cast as the stochastic optimal control problem of taking actions in each state in the corresponding SMDP such that the average reward rate is maximized. Our model utilizes a biologically plausible learning algorithm to solve this problem. The simulation results show that at the beginning of learning the model choses high values of decision threshold which lead to sub-optimal performance. With experience, however, the model learns to lower the value of decision thresholds till finally it finds the optimal values.

The article proposes a method for the synthesis of a neural controller for closed-loop systems with linear objects. The scientific novelty of the proposed method lies in the fact that the neural controller, to the input of which the object state vector is fed, must be trained to stabilize in one of the possible desired values, and to ensure regulation in other desired values. For objects with an inaccessible state vector, it is possible to use the estimation vector of the object state vector. It is proposed to proportionally decrease/increase the signal of the state vector and increase/decrease the control signal formed by the neural regulator. Also, other advantages of the proposed method include: 1) the absence of the need for training on several desired values, which greatly simplifies and accelerates the training of the neural network, and also eliminates control errors in the range of values for which the neural controller was not trained; 2) the possibility of learning from an initially unstable state of a closed-loop system. The proposed method for the synthesis of a neural controller for a closed-loop system with a linear object was tested on the example of the synthesis of a controller for an object 1/s 3, which is unstable. A neural network is used as a regulator, which is proposed to be trained using one of the reinforcement learning methods (in the article, the Deterministic Policy Gradient method allowed us to obtain the best results). The resulting graphs of transient processes allow us to conclude about its successful application. The article ends with conclusions and considerations about further lines of research, which include the quality of the transient process and the possibility of adjusting it by changing the reward function, which will allow setting the graphs of transient processes.

Biosensors are tiny wireless medical devices which are attached or implanted into the body of a human being or animal to monitor and control biological processes. They are distinguished from conventional sensors by their biologically derived sensing elements. Biosensors generate heat when they transmit their measurements and their temperature rises when recharged by electromagnetic energy. These phenomena translate to a temperature increase in the tissues surrounding the biosensors. If the temperature increase exceeds a certain threshold, the tissues might be damaged. In this paper, we discuss the problem of finding an optimal operating policy for a rechargable biosensor under a strict maximum temperature increase constraint. This problem can be formulated as a Markov decision process with an average reward criterion. The solution is an optimal policy that maximizes the average number of samples which can be generated by the biosensor while observing the constraint on the maximum safe temperature level. Due to the exponential nature of the problem, a heuristic policy is proposed. The performance of the policies is studied through simulation. A greedy policy is used as a baseline for comparison.