Self Training Autonomous Driving Agent

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

The essence of designing a good paging strategy is to incorporate the user mobility characteristics in a predictive mechanism that reduces the average paging cost with as little computational effort as possible. In this scope, we introduce a novel paging scheme based on the concept of reinforcement learning. Learning endows the paging mechanism with the predictive power necessary to determine a mobile terminal's position, without having to extract a location probability distribution for each specific user. The proposed algorithm is compared against a heuristic randomized learning strategy akin to reinforcement learning, that we invented for this purpose, and performs better than the case where no learning is used at all. It is shown that if the user normally moves only among a fraction of cells in the location area, significant savings can be achieved over the randomized strategy, without excessive time to train the network. Copyright © 2003 John Wiley & Sons, Ltd.

The essence of designing a good paging strategy is to incorporate user mobility characteristics in a predictive mechanism that reduces the average paging cost with as little computational effort as possible. We introduce a novel paging scheme based on the concept of reinforcement learning. Learning endows the paging mechanism with the predictive power necessary to determine a mobile terminal's position, without having to extract a location probability distribution for each specific user. The proposed algorithm is compared against a heuristic randomized learning strategy akin to reinforcement learning, that we invented for this purpose and performs better than the case where no learning is used at all. It is shown that if the user normally moves only among a fraction of cells in the location area, significant savings can be achieved over the randomized strategy, without excessive time to train the network.

Tuning the seismic control systems in order to achieve optimal performance is a challenging area due to the system and disturbance uncertainties. Although, model uncertainties, process time delay, and actuator dynamics can be considered as typical uncertainties, the main source of uncertainty in a seismic control system comes from the aleatory nature of earthquake disturbances. In this case, tuning of the control system based on a given seismic record may not necessarily result in optimal performance for other earthquakes. In this paper, a methodological approach is proposed for online control of active structural control systems considering seismic uncertainties. For this purpose, the concept of reinforcement learning is utilized for online tuning of an active mass drive system. The controller comprises a gain-scheduling fuzzy proportional derivative controller whose gains are tuned via an online reinforcement learning algorithm. Moreover, in order to tackle the time delays, a dynamic state predictor is utilized in conjunction with the proposed controller. To evaluate the performance of proposed controller, according to an assumed site hazard, thousand ground motion records are generated and clustered based on their spectral features using a fuzzy clustering approach. Finally, the controller is implemented in a laboratory-scale structure, and its performance is examined in the presence of the cluster centers and some real seismic records simulated on a shake table. The test results reveal successful performance of the proposed controller in tackling a wide range of seismic disturbances in the presence of time delay.

Mobile Edge Computing (MEC) has emerged as a key paradigm to support real- time, resource-intensive, and latencycritical applications in 5G and beyond. However, the dynamic nature of workloads, device heterogeneity, and energy constraints introduce significant challenges for effective load balancing and energy-aware resource management. Artificial Intelligence (AI)driven methods, including Deep Reinforcement Learning (DRL), Markov Decision Processes (MDP), and Probabilistic Cellular Automata (PCA), have shown potential in addressing these issues through adaptive decision-making and multi-objective optimization. This survey systematically reviews state-of-the-art AI- driven approaches for MEC optimization, highlighting their strengths, limitations, and applicability across diverse IoT, vehicle, and smart city use cases. Unlike prior surveys, this work provides (i) a structured methodology for literature selection and classification, (ii) comparative evaluation across latency, energy efficiency, scalability, and security dimensions, and (iii) a critical analysis of why certain approaches succeeds or fail in specific contexts. The paper also introduces a conceptual framework that synthesizes heuristic, probabilistic, and reinforcement learning paradigms, serving as a foundation for hybrid strategies. Finally, it identifies open challenges and outlines future research directions such as federated DRL, renewable-energy-aware MEC, and cross-layer optimization for 6G systems.

Many robotic tasks, such as mobile manipulation, often require interaction with unstructured environments and are subject to imperfect sensing and actuation. This brings substantial uncertainty into the problems. Reasoningunder this uncertainty can provide higher level of robustness but is computationally significantly more challenging. More specifically, sequential decision making under motion and sensing uncertainty can be formulated in a principled form as a Partially Observable Markov Decision Process (POMDP). Solving POMDPs exactly is computationally intractable due to their exponential complexity with the number of states and the depth of the planning horizon, so called, the curse of dimensionality and the curse ofhistory, respectively. In this work, we propose a novel search-based robust planning framework that sample-efficiently finds a solution with theoretical suboptimalitybounds by leveraging multiple heuristics that are designed using domain knowledge. The main contributions of this work can be summarized as follows: 1) it works with generative models, in the absence of mathematical models, similarly to the reinforcement learning paradigm, but 2) it still exhibits high sample-efficiency by bootstrapping with the domain knowledge in the form of heuristics, and 3) it is empowered with effective guidance of the search toward the goal through the systematic employment of multipleheuristics. Its solution can be returned in an anytime fashion, and converges to the bounded suboptimality with theoretical guarantees. We validate the proposed frameworks through simulation and robot experiments,which includes 2D rover navigation, PR2 parts assembly, and full size mobile manipulator truck unloading tasks. The truck unloading task is especially interesting, where the custom-built robot is to unload up to several thousands of boxes from a truck. It is a challenging real-world manipulation problem in continuous space with excessively high uncertainty, and we demonstrate the efficacy of our approach in such a domain.

Closed-loop control of anesthesia and mean arterial pressure using reinforcement learning

In computer science, reinforcement learning is a powerful framework with which artificial agents can learn to maximize their performance for any given Markov decision process (MDP). Advances over the last decade, in combination with deep neural networks, have enjoyed performance advantages over humans in many difficult task settings. However, such frameworks perform far less favorably when evaluated in their ability to generalize or transfer representations across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving “deep transfer” characteristic of human behavior has been elusive. Such transfer typically requires discovery of abstractions that permit analogical reuse of previously learned representations to superficially distinct tasks. Here, we demonstrate that abstractions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such reward-predictive representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. These results contrast with those using abstractions that myopically maximize reward in any given MDP and motivate further experiments in humans and animals to investigate if neural and cognitive systems involved in state representation perform abstractions that facilitate such equivalence relations.

Reinforcement Learning (RL) Markov Decision Processes is studied with an emphasis on the well-studied exploration problem. We first formulate and discuss a definition of “efficient” algorithms that is termed Probably Approximately Correct (PAC) in RL. Next we provide general sufficient conditions for such an algorithm that applies to several different modeling assumptions. The conditions can be used to demonstrate that efficient learning is possible in finite MDPs, with either a model-based or model-free approach, in factored MDPs, and in continuous MDPs with linear dynamics. In the reinforcement-learning (RL) problem (Sutton & Barto 1998), an agent acts in an unknown or incompletely known environment with the goal of maximizing an external reward signal. In the most standard mathematical formulation of the problem, the environment is modeled as a Markov Decision Process (MDP) and the goal of the agent is to obtain near-optimal discounted return. Over the years, many algorithms have been proposed for this problem, but analyses of their performances have been relatively scarce. In fact, until recently, most theoretical guarantees have been that certain algorithms will discover an optimal policy in the limit, after an infinite amount of experience. In contrast, several attempts have been made to study “Probably Approximately Correct” or PAC-MDP algorithms, which exhibit near-optimal behavior in polynomial time and experience. This paper discusses several extensions of those results. We present a theorem that provides sufficient conditions for an algorithm to be PAC-MDP. We examine these conditions and show how they can be applied to prove that efficient learning is possible in three interesting scenerios: finite MDPs (i.e. the “Tabular case”),

Development and Evaluation of a Smart Charging Strategy for an Electric Vehicle Fleet Based on Reinforcement Learning

Task offloading in distributed computing involves complex trade-offs among delay, scalability, cost, and resource utilization. Cloud platforms face long communication delays, while edge nodes have constrained capacity. Static, rule-based schedulers cannot adapt to fluctuating loads or per-instance heterogeneity. Similarly, Reinforcement Learning (RL) schemes typically address only a single layer or assume homogeneous servers, overlooking the hierarchical and multi-instance nature of deployments. To address these challenges, we introduce a server-aware Proximal Policy Optimization (PPO) framework that performs fine-grained offloading across a three-tier (Edge, Regional, Cloud), multi-instance architecture. We formulate offloading as a Markov Decision Process whose state vector includes per-instance delay, CPU/memory utilization, network congestion, cost, and energy metrics. The PPO agent learns to offload tasks to the best server in real time. Our developed RegionalEdgeSimPy simulation shows that PPO agent makes optimal offloading choices for over 90% of tasks, keeping each server near; however, below a 70% utilization. This optimized decision making drives up to 66.9% delay reduction, 78.6% energy savings, and 47.8% cost reductions relative to cloud-only and edge-only baselines.

This paper investigates the feasibility of applying\nreinforcement learning (RL) concepts to industrial process\noptimisation. A model-free action-dependent adaptive critic\ndesign (ADAC), coupled with sequential learning neural\nnetwork training, is proposed as an online RL strategy\nsuitable for both modelling and controller optimisation. The\nproposed strategy is evaluated on data from an industrial\ngrinding process used in the manufacture of disk drives.\nComparison with a proprietary control system shows that\nthe proposed RL technique is able to achieve comparable\nperformance without any manual intervention.

Mobile ad-hoc networks (MANETs) are decentralized and self-organizing communication systems. They have become pervasive in the current technological framework. MANETs have become a vital solution to the services that need flexible establishments, dynamic and wireless connections such as military operations, healthcare systems, vehicular networks, mobile conferences, etc. Hence it is more important to estimate the trustworthiness of moving devices. In this research, we have proposed a model to improve a trusted routing in mobile ad-hoc networks by identifying malicious nodes. The proposed system uses Reinforcement Learning (RL) agent that learns to detect malicious nodes. The work focuses on a MANET with Ad-hoc On-demand Distance Vector (AODV) Protocol. Most of the systems were developed with the assumption of a small network with limited number of neighbours. But with the introduction of reinforcement learning concepts this work tries to minimize those limitations. The main objective of the research is to introduce a new model which has the capability to detect malicious nodes that decrease the performance of a MANET significantly. The malicious behaviour is simulated with black holes that move randomly across the network. After identifying the technology stack and concepts of RL, system design was designed and the implementation was carried out. Then tests were performed and defects and further improvements were identified. The research deliverables concluded that the proposed model arranges for highly accurate and reliable trust improvement by detecting malicious nodes in a dynamic MANET environment.

Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex

Reinforcement Learning (RL) is a machine learning methodology that develops the capability to make sequential decisions in intricate issues using trial-and-error techniques. RL has become increasingly prevalent for decision-making and control tasks in diverse fields such as industrial processes, biochemical systems and energy management. This review paper presents a comprehensive examination of the development, models, algorithms and practical uses of RL, with a specific emphasis on its application in process control. The study examines the fundamental theories, methodology and applications of RL, classifying them into two categories: classical RL such as such as Markov decision processes (MDP) and deep RL viz., actor critic methods. RL is a topic of discussion in multiple process industries, such as industrial chemical process control, biochemical process control, energy systems, wastewater treatment and the oil and gas sector. Nevertheless, the paper also highlights challenges that hinder its larger acceptance, including the requirement for substantial computational resources, the complexity of simulating real-world settings and the challenge of guaranteeing the stability and resilience of RL algorithms in dynamic and unpredictable environments. RL has demonstrated significant promise, but more research is needed to fully integrate it into industrial and environmental systems in order to solve the current challenges. Abbreviations: AC: Actor critic; AI: Artificial intelligence; ANN: Artificial neural networks; A3C: Asynchronous advantage actor critic; CRL : Classical Reinforcement learning; CV : Controlled variable; DDPG : Deep deterministic policy gradient; DQN: Deep Q network; DRL: Deep reinforcement learning; DP: Dynamic programming; FOMDP: Fully observable Markov decision process; GRU: Gated recurrent unit; LQR: Linear quadratic regulator; LSTM: Long short-term memory; ML: Machine learning; MV : Manipulated variable; MC: Monte Carlo; MDP: Markov decision process; MPC: Model predictive controller; MIMO: Multi input multi output; PG: Policy gradient; PID: Proportional integral derivative; PPO: Proximal policy optimisation; RL: Reinforcement learning; PPO: Proximal policy optimisation; SAC: Soft actor critic; SISO: Single input single output; TD: Temporal difference; TRPO: Trust region policy optimisation; TD3: Twin delayed deep deterministic policy gradient.