Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

Pneumatic Artificial Muscle (PAM) is an attractive device to be used as an actuator for humanoid robots because of its high power-to-weight ratio and good flexibility. However, both the modeling and the controlling of PAM-driven robots are challenging due to the high nonlinearities of a PAM's air pressure dynamics and its mechanical structure. This paper focuses on applying Reinforcement Learning (RL) to the control of a PAM-driven robots without our knowledge of its model. We propose a new RL algorithm, Local Update Dynamic Policy Programming (LUDPP), as an extension of Dynamic Policy Programming (DPP). This algorithm exploits the nature of smooth policy update of DPP to considerably reduce the computational complexity in both time and space: at each iteration, this algorithm only updates the value function locally throughout the whole state-action space. We applied LUDPP to control one finger (2 DOFs with a 12-dimensional state-action space) of Shadow Dexterous Hand, a PAM-driven humanoid robot hand. Experimental results suggest that our method can achieve successful control of such a robot with a limited computational resource whereas other conventional value function based RL algorithms (DPP, LSPI) cannot.

Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process

Open AccessOpen Access licenseAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail Go to SectionOpen AccessOpen Access license HomeStochastic SystemsVol. 3, No. 2 Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct SearchMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluPublished Online:15 Nov 2013https://doi.org/10.1287/10-SSY020AbstractIn this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation. Back to Top Next FiguresReferencesRelatedInformationCited byInpatient Overflow: An Approximate Dynamic Programming ApproachJ. G. Dai, Pengyi Shi16 May 2019 | Manufacturing & Service Operations Management, Vol. 21, No. 4A Unified Framework for Optimization Under UncertaintyWarren B. Powell4 November 2016The Minimum Expected Penalty Relocation Problem for the Computation of Compliance Tables for Ambulance VehiclesThije van Barneveld21 April 2016 | INFORMS Journal on Computing, Vol. 28, No. 2Discrete Optimization Models for Homeland Security and Disaster ManagementLaura Albert McLay26 October 2015Simulation-Based Approximate Policy Iteration with Generalized Logistic FunctionsAntoine Sauré, Jonathan Patrick, Martin L. Puterman28 September 2015 | INFORMS Journal on Computing, Vol. 27, No. 3Technical Note—Trading Off Quick versus Slow Actions in Optimal SearchSteven M. Shechter, Farhad Ghassemi, Yasin Gocgun, Martin L. Puterman6 March 2015 | Operations Research, Vol. 63, No. 2A Bound on the Performance of an Optimal Ambulance Redeployment PolicyMatthew S. Maxwell, Eric Cao Ni, Chaoxu Tong, Shane G. Henderson, Huseyin Topaloglu, Susan R. Hunter16 July 2014 | Operations Research, Vol. 62, No. 5 Volume 3, Issue 2December 2013Pages 322-633 Article Information Metrics Information Received:November 01, 2010Published Online:November 15, 2013 Copyright © 2013, The author(s)Cite asMatthew S. Maxwell, Shane G. Henderson, Huseyin Topaloglu (2013) Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct Search. Stochastic Systems 3(2):322-361. https://doi.org/10.1287/10-SSY020 PDF download

We present PRM-RL, a hierarchical method for long-range navigation task completion that combines sampling-based path planning with reinforcement learning (RL). The RL agents learn short-range, point-to-point navigation policies that capture robot dynamics and task constraints without knowledge of the large-scale topology. Next, the sampling-based planners provide roadmaps which connect robot configurations that can be successfully navigated by the RL agent. The same RL agents are used to control the robot under the direction of the planning, enabling long-range navigation. We use the Probabilistic Roadmaps (PRMs) for the sampling-based planner. The RL agents are constructed using feature-based and deep neural net policies in continuous state and action spaces. We evaluate PRM-RL, both in simulation and on-robot, on two navigation tasks with non-trivial robot dynamics: end-to-end differential drive indoor navigation in office environments, and aerial cargo delivery in urban environments with load displacement constraints. Our results show improvement in task completion over both RL agents on their own and traditional sampling-based planners. In the indoor navigation task, PRM-RL successfully completes up to 215 m long trajectories under noisy sensor conditions, and the aerial cargo delivery completes flights over 1000 m without violating the task constraints in an environment 63 million times larger than used in training.

This paper treats a class of finite-stage dynamic programming (DP) problems whose state space is an interval in the one-dimensional Euclidean space R1. By introducing a notion of “reward space” we can define, in this class, an inverse problem (called the inverse DP) to a given DP (called the main DP), because of the monotonicity and continuity of the reward functions and state transformations. Roughly speaking the inverse DP to the main DP is defined by replacing “max”, “reward functions” and “state transformations” with “min”, “inverse (in some sense) to the state transformations” and “inverse (in some sense) to the reward functions”, respectively, by replacing “terminal reward function” with “inverse to itself”, and by alternating “reward spaces” with “state spaces”, but by letting “action spaces” remain as they are. According to the monotonicity of the reward functions and state transformations we clarify the relation between the main DP and the inverse DP separately for four cases. Our main result is stated as follows. The pair of optimal reward functions and optimal policy for the main DP characterizes, in an inverse sense, the pair of optimal reward functions and optimal policy for the inverse DP, and vice versa (Section 2). Further we show that our DP represents a mathematical programming problem which has the property “recursiveness with monotonicity” in [5] and conversely such a problem can also be represented by our DP (Section 3). The last section is devoted to the illustration of several examples.

This paper investigates the computation of transient-optimal policies in discrete dynamic programming. The model, is quite general: it may contain transient as well as nontransient policies. and the transition matrices are not necessarily substochastic. A functional equation for the so-called transient-value-vector is derived and the concept of superharmonicity is introduced. This concept provides the linear program to compute the transientvalue-vector and a transient-optimal policy. We also discuss the elimination of suboptimal actions, the solution of problems with additional constraints, and the computation of an efficient policy for a multiple objective dynamic programming problem.

This study explores the potential for artificial agents to develop core consciousness, as proposed by Antonio Damasio's theory of consciousness. According to Damasio, the emergence of core consciousness relies on the integration of a self model, informed by representations of emotions and feelings, and a world model. We hypothesize that an artificial agent, trained via reinforcement learning (RL) in a virtual environment, can develop preliminary forms of these models as a byproduct of its primary task. The agent's main objective is to learn to play a video game and explore the environment. To evaluate the emergence of world and self models, we employ probes–feedforward classifiers that use the activations of the trained agent's neural networks to predict the spatial positions of the agent itself. Our results demonstrate that the agent can form rudimentary world and self models, suggesting a pathway toward developing machine consciousness. This research provides foundational insights into the capabilities of artificial agents in mirroring aspects of human consciousness, with implications for future advancements in artificial intelligence.

Optimal load clipping with time of use rates

Planning by Incremental Dynamic Programming

The computer chess is an important research direction of artificial intelligence. Amazon chess game is an extremely complicated strategy game and the search algorithm relies heavily on the evaluation function. The design of evaluation function faces great challenges. This process requires a comprehensive consideration of the characteristics of the board and build up a series of complex formulas, which requires a lot of human experience to support. The author tries to apply the reinforcement learning method to the design process of Game of the Amazons evaluation function. Using the self-game generated data sets and reward signals to stimulates network, it could learn how to evaluate the situation of chess board without human experience. Then this neural network can be used as an evaluation function. Finally, through battling with the game of Amazon chess game using traditional evaluation function, it is verified that the evaluation function design method proposed in this paper can reduce the human experience requirement in the design process of Game of the Amazons evaluation function and effectively evaluate the situation.

Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning

We previously proposed a method (1998) for acquiring a good evaluation function of the game Othello based on min-max reinforcement learning. In the previous work, we employed a gradient descent method for training the normalized Gaussian network (NGnet) that represents the Othello's evaluation function. However, a lot of games were required to train the NGnet. In this study, we employ the previously proposed online EM algorithm to train the NGnet. The online EM algorithm converges faster than the gradient descent method, and it is suitable for dynamic environments, such as in reinforcement learning tasks. In this article, we introduce a new architecture that is composed of a certain number of NGnets to represent the evaluation function. In this architecture, each of the NGnets is independently trained by the online EM algorithm. Our experiments show that a good evaluation function can be obtained by this new architecture through a smaller number of training games than by the previous scheme.

As an efficient mode of public transportation, urban rail transit promotes sustainable development by reducing energy consumption and improving service quality. Numerous studies have been conducted on this subject. However, many existing methods struggle to adapt to environmental changes due to the complexity of actual line conditions. Therefore, based on deep deterministic policy gradient (DDPG) and genetic algorithm (GA), a new optimization algorithm GA-DDPG is proposed in this paper. This algorithm utilizes DDPG for continuous action control of trains and employs GA to optimize the noise generation hyperparameter of DDPG, thereby enhancing the learning and adaptive capabilities of the algorithm. Firstly, a reference system is established based on expert knowledge to ensure safe train operation and guide the training of the algorithm model. Then, under the intervention of reference system, the GA-DDPG is used to optimize and adjust the train operation control strategy to achieve a balance in punctuality, energy efficiency, and comfort. Finally, simulation experiment shows that GA-DDPG achieves at least 5.2% energy savings and 51.2% passenger comfort compared to DDPG. More experiments further demonstrate that GA-DDPG possesses better adaptability and robustness than DDPG when the environment changes.

The autonomous UAV (unmanned aerial vehicle) navigation has recently gained an increasing interest from both academic and industrial sectors due to its potential uses in various fields and especially, the need for social distancing during the pandemic. Many works have adopted a deep reinforcement learning (RL) method with experience replay called deep deterministic policy gradient (DDPG) to control the motion of UAV, and gain high accuracy results in static and simplified environments. However, they are still far from being ready for real world adoption in that the UAVs have to operate under complex and dynamic conditions. We also found that using only DDPG makes the learning process prone to oscillation and is inefficient for tasks having high dimensional action-state spaces. Furthermore, the goal reward mechanism in traditional reward functions brings a bias to the state, which resembles the one at the goal area and leads to erroneous action selection. To get closer to being ready for real world adoption, we proposed a novel method that enables UAVs to be capable of handling motion control in realistic environments. The first component of our proposed method is point cloud data (PCD) simplification with truncated icosahedron structure which converts enormous PCD into a few essential data points. In the second component of our method, we replace the traditional goal reward mechanism with a new mechanism called Augmentative Backward Reward (ABR) function to dispense the goal reward to transitions proportionately to its participation. By integrating simplified PCD and ABR, we achieved significantly better results when compared with using only the-state-of-the-art, TD3. In addition, we tested the proposed method with another navigation task, BipedalWalkerHardcore, a testbed for RL, and the result is still better and steadier than of TD3. These results indicate that the proposed method is robust.

SummaryRedundant multithreading (RMT) is an effective thread‐level replication method to improve the reliability requirements of applications. Although it significantly improves the robustness of applications, it comes with additional performance overhead since the redundant threads might share the same core resources. In our previous study [Efficient selective replication of critical code regions for SDC mitigation leveraging redundant multithreading.J Supercomput2021;77:14130‐14160], we presented an efficient software‐level RMT approach, where we execute the most critical code regions with three threads to correct errors. In this study, we focus on further improving the performance of our software‐level RMT method by presenting a set of different thread‐to‐core mapping alternatives. We provide different static mapping methods, which require preliminary information about the applications, such as execution time, instruction‐per‐cycle (IPC), or cache usage patterns, and a set of dynamic mapping methods, which map threads to cores dynamically at runtime without requiring any additional information. The dynamic mapping methods decide which threads are mapped to which cores at each scheduling point based on the IPC, cache miss rate, or cache access values of each thread as well as each core. Experimental results show that the dynamic mapping method, which maps threads to cores based on IPC values, outperforms all other static and dynamic methods. It also outperforms our baseline model, where the operating system handles the thread‐to‐core mappings by 8%, 7%, and 20% based on average speedup, harmonic speedup, and mean slowdown metrics.