Acting, Planning, and Learning

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

The paper presents the study of the educational context and its characteristics to achieve the most effective learning process.The aim of the research is to examine the perception of physical space in the traditional classroom and virtual space in online learning. The study of this issue is based on the psychological approach to the perception of virtual space as a model of physical space and on understanding the importance of this research area due to the development of distant forms of learning: though, the problems of virtual space functioning in the perception of students and its impact on learning efficiency still remain unexplored. Meanwhile, understanding the mechanism of space perception can help to adapt the learning process to the form of learning in choice: learning in a traditional classroom, where students work in a real physical space, learning in a virtual environment or in a hybrid form.Materials and methods. The theoretical basis for comparing the characteristics of physical and virtual space is the studies that focus on the importance of spatial organization in the process of learning. The psychological approach to the perception of space makes it possible to rely on the mechanism of transferring the physical characteristics of space into a virtual environment and to determine the possibility of comparison, primarily in an axiological sense. The study was carried out in the context of university education. Students of a higher educational institution were offered a questionnaire, the results of which were processed using a statistical method. Quantitative trends were then interpreted in the application to the learning process. The experiment was carried out in three stages (questionnaire, statistical processing, interpretation of the results), each of which revealed new aspects of the problem under study.Results. The study shows that there is a metaphorization mechanism that allows the learner to perceive the space of the virtual classroom in the same terms as the physical space of the traditional classroom. At the same time, the comparison of statistical data indicates the obvious differences in the perception of the student's space in the traditional and virtual classrooms. This is especially true for such parameters as openness, freedom of movement and the ratio of participants in the educational space, which indicates different learning conditions that determine the choice of teaching methods and techniques in the traditional context and in the virtual educational environment. Discussion of the results of statistical analysis allows us to draw conclusions about the difference of educational potential of the virtual and physical classroom spaces, the importance of socialization and various methods of its creation in the educational process.Conclusion. The conducted research shows the need for further study of space in the educational context, the need to take into account its parameters for developing virtual courses. Also, the results of the study show the importance of spatial characteristics for the success of learning, for creating a comfortable educational environment. The data obtained can be used in the development and improvement of online courses; it will help to avoid mistakes when in planning.

So far, agent research can be broadly categorized into two areas: intelligent agents and mobile agents. Intelligent agent research has grown from a background of AI, humancomputer interaction, and autonomous systems, while mobile agents research stems from the areas of distributed computing and networked systems. Intelligent agents research addresses intelligent control, planning, domain modeling, user modeling, perception, interagent cooperation, and adaptivity. Intelligent agents can take on many different roles, ranging from representing a human user’s interests to actively collaborating with users and each other. Intelligent agents are being designed to tackle nearly every tedious, non-creative activity that people perform (e.g., Web searches), as well as many more abstract functions (e.g., entertainment). Major issues in intelligent agents research include knowledge representation and communication, architectures for intelligent autonomy, mixed-initiative behavior, collaboration techniques, and learning. Mobile agents address physical communication, agent mobility, tracking and locating mobile agents, security and reliability of agents, etc. Mobile agents are suited for optimizing communication ‐ moving the action towards the source of data ‐ and thereby avoiding the need for huge data transfers. This is beneficial in data-intensive applications such as data mining. Mobility can also improve reliability in certain cases. In the field of electronic commerce, mobile agents naturally reflect the characteristics of buyer and the seller, effectively playing the roles of traveling salesman or broker. They can also be applied for easier software customization, software distribution, and to represent disconnected users. The main issues are in security, providing the infrastructure on the Web (naming, locating, controlling, and messaging) and improving survivability. The biggest challenge for all new agent technology is developing successful applications. For this minitrack, we selected two characteristics of agents that can span the diverging areas of intelligent and mobile agents: mobility and communication. The accepted papers reflect the research issues described in this summary. The minitrack is organized in three sessions. The first session explores the coordination of multiple intelligent agents using dramatically different communication schemes. Andersson and Ygge present an efficient market-based approach to managing very large scale groups of agents engaged in coordination via producer/ consumer transactions. Their CoTree algorithm allows largescale markets to operate efficiently with sparse communication, rapid updates as agent preferences change, and minimal data caching requirements on massively distributed systems. In contrast, Nelson, Penner, and Soken discuss the Interaction Society model of coordination, in which relatively small-scale groups of knowledge-intensive agents collaborate using complex communication about discrete tasking and information requirements. Their model for inter-agent communication and interactions with human users is being applied to build a multi-agent advising system for coordinating military Search and Rescue operations. Finally, Barbuceanu discusses the role of obligations and interdictions in the context of coarse-grain agent interactions. By explicitly capturing social obligations and the costs associated with violating those constraints, he shows how rational agents can achieve coordinated behavior without necessarily sharing goals. The second session addresses Java-based mobile agent systems. The paper by Covaci, Busse and Zhang, describes how a mobile agent system can be used for network management. Their framework is based on existing solutions such as TMN, and can be extended to SNMP.

Ensuring that AI systems do what we, as humans, actually want them to do is one of the biggest open research challenges in AI alignment and safety. My research seeks to directly address this challenge by enabling AI systems to interact with humans to learn aligned and robust behaviors. The way robots and other AI systems behave is often the result of optimizing a reward function. However, manually designing good reward functions is highly challenging and error‐prone, even for domain experts. Although reward functions are often difficult to manually specify, human feedback in the form of demonstrations or preferences is often much easier to obtain but can be difficult to interpret due to ambiguity and noise. Thus, it is critical that AI systems take into account epistemic uncertainty over the human's true intent. As part of the AAAI New Faculty Highlight Program, I will give an overview of my research progress along the following fundamental research areas: (1) efficiently quantifying uncertainty over human intent, (2) directly optimizing behavior to be robust to uncertainty over human intent, and (3) actively querying for additional human input to reduce uncertainty over human intent.

Advances in Virtual Reality (VR) systems, Intelligent Tutoring Systems and Agent Technology make it possible to design and develop Virtual Training Environments (VTEs), where the trainees can immerse themselves and interact directly with the learning domain. This paper presents the Interaction Specification Workspace (ISW) architecture for the specification and design of virtual environments for training purposes. ISW architecture provides the interaction designer with the capability to specify the training interaction with the virtual environment using the Virtual Reality Multi Flow Graph (VR-MFG) as the underlying interaction specification model. ISW implements a design space, where the processes of interaction specification and design of VTEs take place inside a three-dimensional virtual environment, the objects of which are tools by themselves. Inside this workspace, the designer can associate the abstract objects (the components of the VR-MFG) with “actual” objects of the target virtual environment (kept in a “world” database) and apply a number of agent templates with training capabilities.

Virtual reality (VR) entails the use of advanced technologies, including computers and various multimedia peripherals, to produce a simulated (that is, virtual) environment that users perceive as comparable to real world objects and events. In recent years, virtual reality technologies have begun to be used as an assessment and treatment tool in occupational therapy, in part because of the ability to create environments that provide patients with opportunities to engage in meaningful, purposeful tasks that are related to real-life interests and activities. The objective of this study was to determine the suitability and feasibility of using a PC-based, non-immersive, VR system (that is, a system in which the user has a reduced sense of actual presence in and control over the simulated environment) for training individuals with unilateral spatial neglect to cross streets in a safe and vigilant manner. A virtual environment, consisting of a typical city street, was programmed using Superscape's 3D-Webmaster, a 3D web-authoring tool. Twelve subjects, aged 55 to 75 years, participated in the initial feasibility study and, to date, a further eight subjects have participated in the intervention study. Six of the initial subjects and all eight of the intervention subjects had sustained a right hemispheric stroke at least 6 weeks prior to the study. The remaining subjects were healthy age-matched adults who were independently mobile and had no difficulty in crossing streets. The results show that this virtual environment was suitable in both its cognitive and motor demands for the targeted population and indicate that the virtual reality training is likely to prove beneficial to people who have difficulty with crossing streets. The generalizability of these results, and recommendations regarding the use of virtual reality as an occupational therapy intervention, must be substantiated by further studies using a range of VR platforms with people with different cognitive and motor disabilities.

Non-Player Characters (NPCs) are a vital part of computer games. Their behaviour is usually pre determined by the developers before a computer game is released. While these NPCs exist in a virtual environment in which they aim to achieve their objectives, these objectives might involve certain actions which should be both believable and add realism to games. In internet gameplay, these may range from navigation of the environment, attack and defense of an objective or assisting other NPCs or human players to achieve their goals. Various forms of decision making process are now implemented to provide believability and realism to games. These techniques range from reinforcement learning, imitation learning, supervised and unsupervised learning for online and offline games. While these are widely implemented, a new genre of realism is being added to games especially first person shooter games, namely Smart Terrains with virtual destructive environments. Smart Terrain has the potential to improve the realism of games and the behavior of the NPC in games such as first person shooter games where gameplay is fast paced. The potential issue would be if NPCs are dynamic enough to recognize changes in the virtual environment or if the virtual environment provides the required information to assist the NPCs with their objectives or tasks. This paper presents an overview of Smart Terrain in enhancing Online Tactic Agent decision making.

In this paper, an innovative approach to controlling temperature was made. Using Unity as our chosen platform, we crafted a virtual environment, that replicates the real-world temperature change dynamics. For this, a virtual environment mimics an enclosed area with a virtual heat source. Additionally, this environment’s area and air density were also taken into consideration. Then, a virtual sensor and cooler were added for detecting and adjusting the environment temperature. Finally, the ML agent package was integrated for controlling temperature with RL. This integration consisted of adding a script that inherits from the agent class and has environment, sensor, and cooler as variables. In each epi-sode, there was a time duration in which both environment's temperature change was calculated and the reward was set to be given only when the environment's temperature was at a certain range. The empirical findings show the efficiency of our approach, show-casing the system's ability to adapt and optimize temperature regulation parameters over successive iterations. By successfully integrating Unity and ML agents, our study under-scores the transformative potential of harnessing virtual environments and machine learning for advancing temperature control technologies. Keywords: unity, ml agents, temperature control, virtual environment, reinforcement learning.

The rapid advancement of autonomous robots in domains such as healthcare, transportation, and disaster response necessitates robust ethical decision-making frameworks to mitigate risks associated with uncertain environments. This research delves deeply into the application of reinforcement learning (RL) for enabling "safe" decisions in robots facing multiple risky outcomes. By synthesizing over 80 peer- reviewed sources, including recent 2024-2025 surveys on ethical RL and safe RL benchmarks, it systematically addresses three pivotal research questions: (1) How should a robot choose between two risky paths? (2) Can RL augmented with reward shaping effectively reduce harm in unpredictable environments? (3) How do prominent RL algorithms compare in terms of safety efficacy in robotic applications? Key findings indicate that multi-objective RL (MORL) and constrained policy optimization variants, such as Proximal Policy Optimization (PPO) with Lagrangian constraints, outperform traditional RL in balancing ethical imperatives with task performance, achieving up to 40% reductions in violation rates in simulated benchmarks. Reward shaping, particularly potential-based and ethics-informed variants, accelerates ethical convergence by 3-6x while curbing reward hacking—a common pitfall where agents exploit reward loopholes. A novel student-led simulation in a custom gridworld environment demonstrates that shaped rewards enhance average returns by approximately 12% with near-zero harm incidents over 1,000 episodes. This expanded study proposes an integrated Ethical Safe RL (ESRL) framework, incorporating human-in-the-loop feedback and verifiable constraints, tailored for accessible implementation in educational settings. It underscores persistent challenges like distributional shifts in real-world deployment and calls for interdisciplinary ethics integration. By bridging theoretical surveys with empirical simulations, this work empowers high-school researchers to contribute to responsible AI robotics.

Pain and discomfort are perceptible during many medical procedures. In the past, drugs have been the conventional means to alleviate pain, but in many instances, medications by themselves do not provide optimal results. Current advances are being made to control pain by integrating both the science of pain medications and the science of the human mind. Various psychological techniques, including distraction by virtual reality environments and the playing of video games, are being employed to treat pain. In virtual reality environments, an image is provided for the patient in a realistic, immersive manner devoid of distractions. This technology allows users to interact at many levels with the virtual environment, using many of their senses, and encourages them to become immersed in the virtual world they are experiencing. When immersion is high, much of the user's attention is focused on the virtual environment, leaving little attention left to focus on other things, such as pain. In this way virtual reality provides an effective medium for reproducing and/or enhancing the distractive qualities of guided imagery for the majority of the population who cannot visualize successfully.

Traffic flow optimization in urban environments remains one of the key challenges of modern research, even despite the significant volume of scientific works devoted to this topic. Despite the achievements, this problem still does not have a universal solution that would work effectively in real-world scenarios. One of the main difficulties is the processing of a large array of input data, in particular, traffic data, which constantly comes from sensors installed throughout the urban road network. Traditionally, due to the scale of the task, researchers have focused on the development of systems with localized agents. Such agents usually manage traffic at individual intersections, while their coordination is carried out within the framework of multi-stream agent systems. However, modern approaches take into account the volume and complexity of input data through the use of deep learning methods. In particular, the use of the deep deterministic policy gradient (DDPG) algorithm is proposed, on the basis of which large input data can be processed. As part of the experimental study, a simple intersection model was tested to verify the effectiveness of the approach. The DDPG algorithm performed better in the simple model compared to Q-learning. DDPG provided a reward in the range of 4-4.3 points, while the reward of Q-learning was in the range of 2-4 points. To evaluate the performance of the DDPG approach compared to Q-learning and random timings, the main criterion is the average reward per episode. DDPG and Q-learning achieve similar reward levels, but DDPG shows stable convergence (0.04-0.21 points), while Q-learning remains unstable (0.04-0.43 points). The study of intra-episode performance shows that DDPG achieves improvements mainly closer to the end of the episode. Overall, this algorithm has proven successful for this scenario, and the results obtained can serve as a basis for further improvements and applications in more complex traffic scenarios.

Deep reinforcement learning algorithms have demonstrated significant potential in showcasing robotic capabilities within virtual environments. However, applying DRL for practical robot development in realistic simulators like Webots remains challenging due to limitations in existing frameworks, such as complex dependencies and reliance on unrealistic control paradigms like a ‘supervisor’. This study introduces an open-source framework and a novel pattern-based method designed to facilitate the exploration of robot learning capabilities through reinforcement learning algorithms in specialized virtual testing environments built on Webots. Our approach simplifies setup by avoiding burdensome external package installations and, crucially, removes the dependency on an unrealistic ‘supervisor’ entity, offering a more practical and real-world-aligned solution. Designed to leverage Webots’ realistic simulation capabilities, the proposed method and system are validated through various examples, ranging from the classic inverted pendulum scenario to a production robot utilized in an actual assembly line. The developed code and examples are publicly accessible on GitHub for the deep reinforcement learning research community.

Noisy inputs pose significant challenges for reinforcement learning (RL) agents navigating real-world environments. While animals demonstrate robust spatial learning under dynamic conditions, the mechanisms underlying this resilience remain understudied in RL frameworks. This paper introduces a novel comparative analysis of predecessor feature (PF) and successor feature (SF) algorithms under controlled noise conditions, revealing several insights. Our key innovation lies in demonstrating that SF algorithms achieve superior noise resilience compared to traditional approaches, with cumulative rewards of 2216.88±3.83 (mean ± SEM), even under high noise conditions (σ=0.5) in one-dimensional environments, while Q learning achieves only 19.22±0.57. In two-dimensional environments, we discover an unprecedented nonlinear relationship between noise level and algorithm performance, with SF showing optimal performance at moderate noise levels (σ=0.25), achieving cumulative rewards of 2886.03±1.63 compared to 2798.16±3.54 for Q learning. The λ parameter in PF learning is a significant factor, with λ=0.7 consistently achieving higher λ values under most noise conditions. These findings bridge computational neuroscience and RL, offering practical insights for developing noise-resistant learning systems. Our results have direct applications in robotics, autonomous navigation, and sensor-based AI systems, particularly in environments with inherent observational uncertainty.

Lengthy evaluation times are common in many optimization problems such as direct policy search tasks, especially when they involve conducting evaluations in the physical world, for example, in robotics applications. Often when evaluating solution over a fixed time period, it becomes clear that the objective value will not increase with additional computation time (e.g., when a two-wheeled robot continuously spins on the spot). In such cases, it makes sense to stop the evaluation early to save computation time. However, most approaches to stop the evaluation are problem specific and need to be specifically designed for the task at hand. Therefore, we propose an early stopping method for direct policy search. The proposed method only looks at the objective value at each timestep and requires no problem-specific knowledge. We test the introduced stopping criterion in five direct policy search environments drawn from games, robotics, and classic control domains and show that it can save up to \(75\%\) of the computation time. We also compare it with problem-specific stopping criteria and show that it performs comparably, while being more generally applicable.

This study constructs a digital twin of artificial variable width channel (AVWC) experimental device by 3D modeling, and controls it with reinforcement learning (RL). In recent years, as a measure against the increasing occurrence of river flooding, previous research created a river 3D model based on river experimental model. The river 3D model, evaluated through river health assessments, demonstrated potential for utilization as a digital twin. However, to realize the concept of the digital twin, it is necessary to validate the consistency between physical behaviors in the virtual space and physical phenomena. Therefore, this study focuses on controlling a simplified 3D simulation model using groynes with RL for the development of a digital twin of a river. Through this effort, a foundation will be established for verifying the consistency between the control results of the 3D simulation model and a rigid physical experimental device.