Run Time Assured Reinforcement Learning for Safe Satellite Docking

While reinforcement learning (RL) offers high-performance solutions to complex tasks, its trial-and-error paradigm presents safety concerns when exploring unsafe states may be unacceptable. Safety filtering or run-time assurance (RTA) approaches can be used to monitor the RL agent’s desired control and prevent the agent from taking unsafe actions by satisfying safety constraints during and after training. This study investigates the problem of on-orbit inspection of a chief spacecraft using a deputy spacecraft controlled by an RL agent with safety constraints. Several constraints on the state space are enforced using discrete control barrier functions and an optimization-based RTA system. This assures the safety of the deputy spacecraft and its sensors even with a neural-network-based primary controller. A detailed analysis of RL agent performance for eight separate experiments is presented, including each safety constraint individually, all constraints simultaneously, and no constraints. Results demonstrate that the RL agent can complete the inspection task while adhering to safety constraints in a simulated RL environment. These results also show that the RTA system can aid the RL agent in learning a successful policy over fewer time steps but may lead to higher fuel usage, depending on the constraint(s) enforced.

Increasingly higher demands are being made on the capacity‐limited cognitive capabilities of human operators as they strive to maintain situation awareness (i.e., understanding “what is going on”) and performance in complex tasks. In the current study we asked whether: (a) training administered via a mobile phone‐based app could improve multitasking and (b) improved multitasking in the app would generalize to improved performance and situation awareness in a simulated air traffic control task (ATC). Participants completed the ATC task before and after multiple sessions of app‐based multitasking training or control training. Multitasking on the app improved across training sessions. However, this did not lead to improved performance or situation awareness, or workload reduction, relative to control training on the ATC task. These outcomes indicate that app‐based multitasking training based on repetition of a single training task will not necessarily yield generalizable benefits to human performance in other complex dynamic tasks.

We propose using a statistical-relational model, the Markov Logic Network, for knowledge transfer in reinforcement learning. Our goal is to extract relational knowledge from a source task and use it to speed up learning in a related target task. We show that Markov Logic Networks are effective models for capturing both source-task Q-functions and source-task policies. We apply them via demonstration, which involves using them for decision making in an initial stage of the target task before continuing to learn. Through experiments in the RoboCup simulated-soccer domain, we show that transfer via Markov Logic Networks can significantly improve early performance in complex tasks, and that transferring policies is more effective than transferring Q-functions.

: The proposed research was designed to extend prior AFOSR sponsored research (DeShon, Kozlowski et al., 2004) to model optimal human resource allocation to account for learning, performance, and adaptation for complex and dynamic tasks incorporating individual and team goals. Phase 1 was intended to implement an optimal, multiple-criterion (individual and team goals) reinforcement learning model that would compare human performance to optimal model performance. Phase 2 was intended to extend the model to autonomous decision makers by incorporating into the decision maker via satiation levels on individual and team goals, with learning and performance compared to the optimal model (Phase I) and human benchmarks. Phase 3 was intended to extend the model to encompass adaptation to changes in reward structure (development of an adaptive, model-based reinforcement learning approach that would be compared to standard reinforcement learning and human performance). Funding restrictions limited work to 50% of phase 1 effort (6 of 12 months at 50% of original budget). Funding ceased at 6 months. Initial project efforts were devoted to redesigning and redeveloping our individual-team resource allocation simulation to incorporate features necessary to implement the reinforcement learning model and to evaluate potential implementations of the Q-learning algorithm within the simulation. This report summarizes the intended research contribution and progress up to funding termination.

Reinforcement learning (RL) has shown outstanding performance in handling complex tasks in recent years. Eligibility trace (ET), a fundamental and important mechanism in reinforcement learning, records critical states with attenuation and guides the update of policy, which plays a crucial role in accelerating the convergence of RL training. However, ET implementation on conventional digital computing hardware is energy hungry and restricted by the memory wall due to massive calculation of exponential decay functions. Here, in-memory realization of ET for energy-efficient reinforcement learning with outstanding performance in discrete- and continuous-state RL tasks is demonstrated. For the first time, the inherent conductance drift of phase change memory is exploited as physical decay function to realize in-memory eligibility trace, demonstrating excellent performance during RL training in various tasks. The spontaneous in-memory decay computing and storage of policy in the same phase change memory give rise to significantly enhanced energy efficiency compared with traditional graphics processing unit platforms. This work therefore provides a holistic energy and hardware efficient method for both training and inference of reinforcement learning.

The power system is undergoing a significant shift from fossil fuel‐based electricity generation to inverter‐based renewable energy resources (IBRs), accelerating the transition towards cleaner energy. This transition, however, introduces new challenges for system stability and control. One of the most critical issues is the decline in frequency stability due to reduced system inertia and damping, particularly in isolated or weakly interconnected power systems such as microgrids. Therefore, novel ancillary services capable of delivering fast and effective frequency support that accounts for the dynamic nature of the modern power system are crucial. In this study, we develop a reinforcement learning (RL)‐based control framework to provide fast frequency response (FFR) in a microgrid. The RL‐based controller is trained through continuous interaction with a simulated microgrid environment using the soft actor‐critic (SAC) algorithm, an advanced off‐policy RL technique. To enable efficient RL training, a scalable co‐simulation framework with a real‐time digital environment is employed, allowing a parallel execution of online RL training and microgrid model simulation. The RL training configuration is deployed on the Cordova, Alaska, benchmark microgrid. A detailed evaluation of the trained RL‐based controller demonstrates its ability to deliver efficient and timely frequency support to the microgrid, reducing frequency nadirs by 55.03% and 61.78% in cases with and without under‐frequency load shedding (UFLS). Load impact assessments confirm the controller's robustness under varying loading scenarios, and the computational times during training and testing validate its real‐time applicability for use in microgrids. Additionally, a practical evaluation of energy storage system (ESS) sizing under a 2‐day load profile provides valuable insights into resource considerations for real‐world FFR implementation.

Given that buildings consume approximately 33% of global energy, and HVAC systems contribute nearly half of a building’s total energy demand, optimizing their efficiency is imperative for sustainable energy use. Many existing buildings operate HVAC systems inefficiently, displaying non-stationary behavior. Current reinforcement learning (RL) training methods rely on historical data, which is often obtained through costly modeling or trial-and-error methods in real buildings. This paper introduces a novel reinforcement learning construction framework designed to improve the robustness and learning speed of RL control while reducing learning costs. The framework is specifically tailored for existing office buildings. Applying this framework to control HVAC systems in real office buildings in Beijing, engineering practice results demonstrate: during the data collection phase, energy efficiency surpasses traditional rule-based control methods from the previous year, achieving significantly improved energy performance (a 17.27% reduction) with minimal comfort sacrifices. The system achieves acceptable robustness, learning speed, and control stability. Reduced ongoing manual supervision leads to savings in optimization labor. Systematic exploration of actions required for RL training lays the foundation for RL algorithm development. Furthermore, by leveraging collected data, a reinforcement learning control algorithm is established, validating the reliability of this approach. This construction framework reduces the prerequisites for historical data and models, providing an acceptable alternative for systems with insufficient data or equipment conditions.

Evaluating reward models is a fundamental challenge in Reinforcement Learning (RL), particularly in settings where the reward model is learned or manually designed. The standard paradigm for Reward Model Evaluation (RME) involves training an optimal policy via RL on the given reward model and assessing model quality through the performance of the resulting policy. However, this approach conflates the quality of the reward model with the effectiveness of RL training, and is computationally expensive due to the need for policy optimization. Recent RME methods attempt to circumvent this issue by evaluating reward models directly, without RL, but often rely on impractical assumptions such as access to a ground-truth reward or fail to utilize available supervision in a fine-grained manner. To overcome these limitations, we propose the Policy Preference Alignment Coefficient (PPAC), a novel metric for RME that requires neither RL training nor ground-truth rewards. PPAC first generates a sequence of automatically ranked policy preferences that guarantee monotonic improvement in the policy value, and then quantifies the alignment between these generated preferences and those implied by the candidate reward model. Experimental results across gridworld and continuous control task demonstrate that PPAC yields preference sequences with consistently increasing policy values and outperforms existing metrics in evaluating reward model quality.

Integrated sensing and communications (ISAC) has been envisioned as a pivotal technology in solving the exacerbated spectrum congestion problem. Meanwhile, intelligent reflective surface (IRS) presents a promising avenue for enhancing both communications and sensing in severe obstruction scenarios, such as dense urban area. In this paper, we propose a reinforcement learning (RL)-based IRS-assisted ISAC design that enables simultaneous multi-user downlink communications and multi-target sensing via optimizing the IRS reflection coefficients in unknown high-obstruction scenarios. In order to facilitate the IRS-assisted ISAC design, we first introduce a two-phase method that can achieve precise channel estimation with a small number of pilot signals. Subsequently, we employ the RL to co-design the dual-functional IRS, striving for an optimal performance balance between communications and sensing in unknown environments. Specifically, to address the non-convex problems arising from the RL training, we propose to design the Gramian matrix first, followed by matrix decomposition to obtain the optimal IRS reflection coefficients. Extensive simulations have demonstrated that the two-phase method can accurately estimate the channel, and the IRS-assisted ISAC designed via RL exhibits an excellent dual-functional performance in the unknown environment.

For networks that process 3D data, estimating the orientation and position of 3D objects is a challenging task. This is because the traditional networks are not robust to the rotation of the data, and their internal workings are largely opaque and uninterpretable. To solve this problem, a novel equivariant self-supervised vector network for point clouds is proposed. The network can learn the rotation direction information of the 3D target and estimate the rotational pose change of the target, and the interpretability of the equivariant network is studied using information theory. The utilization of vector neurons within the network lifts the scalar data to vector representations, enabling the network to learn the pose information inherent in the 3D target. The network can perform complex rotation-equivariant tasks after pre-training, and it shows impressive performance in complex tasks like category-level pose change estimation and rotation-equivariant reconstruction. We demonstrate through experiments that our network can accurately detect the orientation and pose change of point clouds and visualize the latent features. Moreover, it performs well in invariant tasks such as classification and category-level segmentation.

Deep Deterministic Policy Gradient (DDPG) algorithm has been successful for state-of-the-art performance in high-dimensional continuous control tasks. However, due to the complexity and randomness of the environment, DDPG tends to suffer from inefficient exploration and unstable training. In this work, we propose Self-Adaptive Double Bootstrapped DDPG (SOUP), an algorithm that extends DDPG to bootstrapped actor-critic architecture. SOUP improves the efficiency of exploration by multiple actor heads capturing more potential actions and multiple critic heads evaluating more reasonable Q-values collaboratively. The crux of double bootstrapped architecture is to tackle the fluctuations in performance, caused by multiple heads of spotty capacity varying throughout training. To alleviate the instability, a self-adaptive confidence mechanism is introduced to dynamically adjust the weights of bootstrapped heads and enhance the ensemble performance effectively and efficiently. We demonstrate that SOUP achieves faster learning by at least 45% while improving cumulative reward and stability substantially in comparison to vanilla DDPG on OpenAI Gym's MuJoCo environments.

Despite the fact that deep reinforcement learning (RL) has surpassed human-level performances in various tasks, it still has several fundamental challenges. First, most RL methods require intensive data from the exploration of the environment to achieve satisfactory performance. Second, the use of neural networks in RL renders it hard to interpret the internals of the system in a way that humans can understand. To address these two challenges, we propose a framework that enables an RL agent to reason over its exploration process and distill high-level knowledge for effectively guiding its future explorations. Specifically, we propose a novel RL algorithm that learns high-level knowledge in the form of a finite reward automaton by using the L* learning algorithm. We prove that in episodic RL, a finite reward automaton can express any non-Markovian bounded reward functions with finitely many reward values and approximate any non-Markovian bounded reward function (with infinitely many reward values) with arbitrary precision. We also provide a lower bound for the episode length such that the proposed RL approach almost surely converges to an optimal policy in the limit. We test this approach on two RL environments with non-Markovian reward functions, choosing a variety of tasks with increasing complexity for each environment. We compare our algorithm with the state-of-the-art RL algorithms for non-Markovian reward functions, such as Joint Inference of Reward machines and Policies for RL (JIRP), Learning Reward Machine (LRM), and Proximal Policy Optimization (PPO2). Our results show that our algorithm converges to an optimal policy faster than other baseline methods.

This work investigates the complications of robotic learning using reinforcement learning (RL). While RL has enormous potential for solving complex tasks its major caveat is the computation cost- and time-intensive training procedure. This work aims to address this issue by introducing a humanlike thinking and acting paradigm to a RL approach. It utilizes model-based deep RL for planning (think) coupled with inverse kinematics (IK) for the execution of actions (act). The approach was developed and tested using a Franka Emika Panda robot model in a simulated environment using the PyBullet physics engine Bullet. It was tested on three different simulated tasks and then compared to the conventional method using RL-only to learn the same tasks. The results show that the RL algorithm with IK converges significantly faster and with higher quality than the applied conventional approach, achieving 98%, 99% and 98% success rates for tasks 1-3 respectively. This work verifies its benefit for use of RL-IK with multi-joint robots.

This study aimed to explore the influence of task complexity, prior knowledge, and gender on individual learning performance in the domains of knowledge and skills learning outcomes. The study examined task complexity (basic vs. advanced skills), and prior knowledge (less vs. more knowledgeable learners), involving 73 undergraduate students of varying genders. The findings indicated that male students outperformed female students in the skills domain for both low-complexity tasks (basic skills) and high-complexity tasks (advanced skills), but this was not the case in the knowledge-and-understanding (recall) domain. In these domains, females scored slightly higher across both complexity levels. In contrast, for high-complexity females' scores were lower, whereas males' scores remained unaffected. Furthermore, females' students with less knowledge generally did not score as highly. This decreased even more for high-complexity tasks. Conversely, males with less knowledge scored higher than their knowledge-and-understanding (recall) domain. Advanced prior knowledge promotes better performance in complex tasks, and there was gender difference.

A propositional structure is set forth that considers: (1) the role of various forms of feedback in eliciting differing motivational orientations toward performance in tasks that reflect ability; (2) the strength of arousal associated with these various motivational orientations as a function of repeated feedback and the age of the performer; and (3) the impact of differing strengths of arousal upon the performance of simple and complex tasks. On the basis of this propositional statement, a series of hypotheses are formulated and tested concerning predicted effects of various forms of feedback upon second, fourth and sixth grade Flemish boys performing simple and complex tasks in a classroom setting. In general, the results are consistent with predictions that the strength of competitive motives, induced by own and other performance feedback, as compared to individualistic motives induced by own feedback and a control condition of no feedback, increase both as a function of trials and the age of the performer, and that these increases in arousal lead to higher levels of performance in simple, but not complex tasks.