Self-Generation of Reward by Sensor Input in Reinforcement Learning

When robots are needed to operate in a dynamic or unpredictable surroundings, this method has been proven success in the numerous robotic uses including navigation, manipulation & the decision-making. Robots can freely negotiate environment, overcome obstacles & the create ideal courses for navigation tasks using reinforcement learning methods. Reinforcement learning improves robots' accuracy & the dexterity in tasks of involving product handling including picking & the placing. Reinforcement learning is a crucial for the decision-making as it helps robots to evaluate possible actions, examine occurrences & adapt with the surroundings. Notwithstanding the great potential of reinforcements learning in robotics, numerous issues still exists including the necessity of significant data for training, the complexity of actual time decision-making & the guarantee of safe learning in the unstructured environments. Notwithstanding these challenges, ongoing research & the development helps to include reinforcement learning into robots, proving promising results in many different fields, from manufacturing to healthcare. By enabling robots to do tasks autonomously, adapt to unforeseen circumstances, and interact with humans in more intuitive and efficient ways, therefore increasing production and enhancing quality of life and setting the future of reinforcement learning in robotics to alter industries.

Robot learning with GA-based fuzzy reinforcement learning agents

In robotics, the ultimate goal of reinforcement learning is to endow robots with the ability to learn, improve, adapt and reproduce tasks with dynamically changing constraints based on exploration and autonomous learning. We give a summary of the state-of-the-art of reinforcement learning in the context of robotics, in terms of both algorithms and policy representations. Numerous challenges faced by the policy representation in robotics are identified. Three recent examples for the application of reinforcement learning to real-world robots are described: a pancake flipping task, a bipedal walking energy minimization task and an archery-based aiming task. In all examples, a state-of-the-art expectation-maximization-based reinforcement learning is used, and different policy representations are proposed and evaluated for each task. The proposed policy representations offer viable solutions to six rarely-addressed challenges in policy representations: correlations, adaptability, multi-resolution, globality, multi-dimensionality and convergence. Both the successes and the practical difficulties encountered in these examples are discussed. Based on insights from these particular cases, conclusions are drawn about the state-of-the-art and the future perspective directions for reinforcement learning in robotics.

In recent years, the recommendation system and robot learning are undoubtedly the two most popular application fields, and the core algorithms supporting these two fields are deep learning based on perception and reinforcement learning based on exploration learning, respectively. How to combine these two fields to better improve the development of the whole machine learning field is the dream of numerous researchers. The Deep Reinforcement Network (DRN) model successfully embedded reinforcement learning into the recommendation system, which provided a good idea for subsequent researchers. However, the disadvantage is also obvious, that is, the DRN model is built for news recommendations, meaning that the DRN model is not transferable, which is also the defect of many current recommendation system models. Meanwhile, the agent learning method adopted by the DRN model is primitive and inefficient. Among many models and algorithms that have emerged in recent years, we use the newly proposed deployment efficiency to measure their comprehensive quality and found that few models focus on both efficiency and performance improvement. To fill the gap of model deployment efficiency neglected by many researchers and to create a model of reinforcement learning agents with stronger performance, we have been exploring and trying to complete research on the Gate Attentional Factorization Machines (GAFM) model. Finally, we successfully integrated the GAFM model and reinforcement learning. The Deep Reinforcement Factorization Machines (DRFM) model proposed in this paper is based on the combination of deep learning with high perception ability and reinforcement learning with high exploration ability, centered on improving the deployment efficiency and learning performance of the model. The GAFM model is modified and upgraded using multidisciplinary techniques, and a new model-based random exploration strategy is proposed to update and optimize the recommendation list efficiently. Through parallel contrast experiments on various datasets, it is proved that the DRFM model surpasses the traditional recommendation system model in all aspects. The DRFM model is far superior to other models in terms of performance and robustness, and also significantly improved in terms of deployment efficiency. At the same time, we conduct a comparative analysis with the latest deep reinforcement learning algorithm and prove the unique advantages of the DRFM model.

Reinforcement learning (RL) problems are hard to solve in a robotics context as classical algorithms rely on discrete representations of actions and states, but in robotics both are continuous. A discrete set of actions and states can be defined, but it requires an expertise that may not be available, in particular in open environments. It is proposed to define a process to make a robot build its own representation for an RL algorithm. The principle is to first use a direct policy search in the sensori-motor space, i.e., with no predefined discrete sets of states nor actions, and then extract from the corresponding learning traces discrete actions and identify the relevant dimensions of the state to estimate the value function. Once this is done, the robot can apply RL: 1) to be more robust to new domains and, if required and 2) to learn faster than a direct policy search. This approach allows to take the best of both worlds: first learning in a continuous space to avoid the need of a specific representation, but at a price of a long learning process and a poor generalization, and then learning with an adapted representation to be faster and more robust.

Common challenges in the area of robotics include issues such as sensor modeling, dynamic operating environments, and limited on-broad computational resources. To improve decision making, robots need a dependable framework to facilitate communication between different modules and the optimal action for real-world applications. The Robotics Operating System (ROS) and Reinforcement Learning (RL) are two promising approaches that help accomplish precise control, seamless integration of sensors-actuators, and exhibit learned behavior. The ROS enables seamless communication between heterogeneous components, while RL focuses on learning optimal behaviors through trial-and-error scenarios. Combining the ROS and RL offers superior decision making, improved perception, enhanced automation, and reliability. This work focuses on investigating ROS-based RL applications across various domains, aiming to enhance understanding through comprehensive discussion, analysis, and summarization. We base our evaluation on the application area, type of RL algorithm used, and degree of ROS–RL integration. Additionally, we provide summary of seminal works that define the current state of the art, along with GitHub repositories and resources for research purposes. Based on the review of successfully implemented projects, we make recommendations highlighting the advantages and limitations of RL techniques for specific applications and environments. The ultimate goal of this work is to advance the robotics field by providing a comprehensive overview of the recent important works that incorporate both the ROS and RL, thereby improving the adaptability of these emerging techniques.

This paper describes the mechanical design and a cognition system for a novel concept-of-camera robotic assistant. The system combines the advantages of intra-abdominal devices and autonomous camera navigation. The robotic assistant is composed of a magnetic intra-abdominal camera robot with two internal cable-driven degrees of freedom and an external robot that handles an external magnet. The intelligence of the robot is implemented in a cognitive architecture based on a long-term memory that stores the robot's knowledge and learning capabilities to improve the robot's behavior. The navigation strategy combines a reactive control based on instrument tracking and a proactive control based on predefined behaviors, depending on the actual state of the task. The robot's learning capabilities include a semantic customization, to adapt the camera's behavior to the surgeon's preferences, and reinforcement learning, to improve the camera navigation strategy. This paper details both the hardware implementation of the system and the software implementation in a robotic operating system architecture. The cognition system and the performance of the cable-driven mechanism have been validated with a set of in vitro experiments. Moreover, the camera robot has been evaluated through an in-vivo experiment in a pig.

Reinforcement learning has become a promising and effective method to solve robots' complex control problems, enabling agents to learn optimal behaviors independently in a dynamic, uncertain, and usually high-dimensional environment. This paper comprehensively reviews the latest progress in applying reinforcement learning technology in robotics, paying special attention to key areas such as path planning, dynamic obstacle avoidance, multi-robot cooperation, and human-computer interaction. By critically analyzing various methods based on reinforcement learning, this paper emphasizes the key progress in improving robot autonomy, decision-making, and adaptability. It also discusses the challenges of deploying reinforcement learning in real-world robot applications, including sample efficiency, security, scalability, and generalization to new environments. In particular, Deep Reinforcement Learning (DRL) shows great potential in enhancing the robot's ability, especially in a complex, unstructured environment. However, challenges such as high computing costs and long training time still need to be addressed. Finally, the paper suggests that future research should focus on the hybrid RL method, improve learning efficiency, combine domain knowledge, and combine RL with other advanced technologies such as computer vision, multi-agent system, and real-time feedback mechanism, to further expand its potential application and increase its influence in the field of robotics.

For autonomous driving, safer travel, and fleet management, Vehicle-to-Vehicle (V2V) communication protocols are an emerging area of research and development. State-of-the-art techniques include machine learning (ML) and reinforcement learning (RL) to adapt modulation and coding rates as the vehicle moves. However, channel state estimations are often incorrect and rapidly changing in a V2V scenario. We propose a combination of input features, including (a) sensor inputs from other parameters in the vehicle, such as speed and global positioning system (GPS), (b) estimation of interference and load for each of the vehicles, and (c) channel state estimation to find the optimal rate that would maximize Quality-of-Service. Our model uses an ensemble of RL-agents to predict trends in the input parameters and to find the inter-dependencies of these input parameters. An RL agent then utilizes these inputs to find the best modulation and coding rate as the vehicle moves. We demonstrate our results through prototype experiments using real data collected from customer networks.

In reinforcement learning, the reward function has a significant impact on the performance of the agent. However, determining the appropriate value of this reward function requires many attempts and trials. Although many automated reinforcement learning methods have been proposed to find an appropriate reward function, their proof is lacking in complex environments such as quadrupedal locomotion. In this paper, we propose a method to automatically tune the scale of the dominant reward functions in reinforcement learning of a quadrupedal robot. Reinforcement learning of the quadruped robot is very sensitive to the reward function, and recent outstanding research results have put a lot of effort into reward shaping. In this paper, we propose an automated reward shaping method that automatically adjusts the reward function scale appropriately. We select some dominant reward functions, arrange their weights in a certain unit, and then calculate their gait scores so that we can select the agent with the highest score. This gait score was defined to reflect the stable walking of the quadrupedal robot. Additionally, quadrupedal locomotion learning requires reward functions of different scales depending on the robot’s size and shape. Therefore, we evaluate the performance of the proposed method on two different robots.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Learning through imitation is a powerful and versatile method for acquiring new behaviors. In humans, a wide range of behaviors, from styles of social interaction to tool use, are passed from one generation to another through imitative learning. Although imitation evolved through Darwinian means, it achieves Lamarckian ends: it is a mechanism for the inheritance of acquired characteristics. Unlike trial-and-error-based learning methods such as reinforcement learning, imitation allows rapid learning. The potential for rapid behavior acquisition through demonstration has made imitation learning an increasingly attractive alternative to manually programming robots. In this chapter, we review recent results on how infants learn through imitation and discuss Meltzoff and Moore's four-stage progression of imitative abilities: (i) body babbling, (ii) imitation of body movements, (iii) imitation of actions on objects, and (iv) imitation based on inferring intentions of others. We formalize these four stages within a probabilistic framework for learning and inference. The framework acknowledges the role of internal models in sensorimotor control and draws on recent ideas from the field of machine learning regarding Bayesian inference in graphical models. We highlight two advantages of the probabilistic approach: (1) the development of new algorithms for imitation-based learning in robots acting in noisy and uncertain environments, and (2) the potential for using Bayesian methodologies (such as manipulation of prior probabilities) and robotic technologies to deepen our understanding of imitative learning in humans.

Peg-in-hole assembly with narrow clearance is a typical robotic contact-rich task in industrial manufacturing. Robot learning allows robots to directly acquire the assembly skills for this task without modeling and recognizing the complex contact states. However, learning such skills is still challenging for robot because of the difficulties in collecting massive transitions data and transferring skills to new tasks, which inevitably leads to low training efficiency. This paper formulated the assembly task as a Markov decision process, and proposed a model accelerated reinforcement learning method to efficiently learn assembly policy. In this method, the assembly policy is learned with the maximum entropy reinforcement learning framework and executed with an impedance controller, which ensures exploration efficiency meanwhile allows transferring skills between tasks. To reduce sample complexity and improve training efficiency, the proposed method learns the environment dynamics with Gaussian Process while training policy, then, the learned dynamic model is utilized to improve target value estimation and generate virtual data to argument transition samples. This method can robustly learn assembly skills while minimizing real-world interaction requirements which makes it suitable for realistic assembly scenarios. To verify the proposed method, experiments on an industrial robot are conducted, and the results demonstrate that the proposed method improves the training efficiency by 31% compared with the method without model acceleration and the learned skill can be transferred to new tasks to accelerate the training for new policies.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Machine Learning (ML) techniques are being employed in bioinformatics with increasing success. However, two problems are still prohibitive for symbolic ML methods: huge amount of data and lack of examples for training purposes. Thus, this paper introduces the use of reinforcement learning (RL), with the objective of dealing with these two drawbacks. Our work proposes and implement a RL method for the Bio Agents system, in order to improve the annotation of biological sequences in genome sequencing projects. Experiments were done with real data from two different genome sequencing projects: Paracoccidioides brasiliensis - Pb fungus and Paullinia cupana - Guaraná plant. To assign reinforcement signals we have used reference genomes with curated annotations that are considered correct, these signals tackle specific databases and alignment algorithms. The results obtained with the inclusion of a RL layer in Bio Agents were better compared with the system without the proposed method. Also, to the best of our knowledge, this is the first attempt to apply RL techniques to annotation in bioinformatics projects.