A structured state–reward compatibility framework for reinforcement learning in resource-constrained cyber-physical systems

This paper presents the design and implementation of a learning controller for the Automatic Generation Control (AGC) in power systems based on a reinforcement learning (RL) framework. In contrast to the recent RL scheme for AGC proposed by us, the present method permits handling of power system variables such as Area Control Error (ACE) and deviations from scheduled frequency and tie-line flows as continuous variables. (In the earlier scheme, these variables have to be quantized into finitely many levels). The optimal control law is arrived at in the RL framework by making use of Q-learning strategy. Since the state variables are continuous, we propose the use of Radial Basis Function (RBF) neural networks to compute the Q-values for a given input state. Since, in this application we cannot provide training data appropriate for the standard supervised learning framework, a reinforcement learning algorithm is employed to train the RBF network. We also employ a novel exploration strategy, based on a Learning Automata algorithm, for generating training samples during Q-learning. The proposed scheme, in addition to being simple to implement, inherits all the attractive features of an RL scheme such as model independent design, flexibility in control objective specification, robustness etc. Two implementations of the proposed approach are presented. Through simulation studies the attractiveness of this approach is demonstrated.

Dynamic holding control to avoid bus bunching: A multi-agent deep reinforcement learning framework

Classification with Costly Features (CwCF) addresses the challenge of balancing feature acquisition costs with classification accuracy. Traditional methods often rely on static feature subsets, leading to suboptimal resource utilization. This paper proposes a novel reinforcement learning (RL) framework enhanced with Transformer architecture to dynamically select features in a sequential manner, optimizing cost-accuracy trade-offs. Our model leverages the Transformer architecture to handle variable-length input sequences, combining embeddings for both feature values and their respective feature identities, while integrating Deep Q-Networks for decision-making. Experiments across synthetic and real-world benchmarks demonstrate that the Transformer-based approach achieves competitive accuracy compared to prior RL methods, though it falls short of state-of-the-art (SotA) models in cumulative return. The results highlight the model’s ability to adaptively gather informative features, emphasizing accuracy over cost minimization. This work brings out the potential of sequence-based architectures in dynamic feature selection and opens discussion for further exploration of Transformer-RL hybrids in cost-sensitive classification tasks.

Nitrous oxide (N2O) emissions from agriculture are rising due to increased fertilizer use and intensive farming, posing a major challenge for climate mitigation. This study introduces a novel reinforcement learning (RL) framework to optimize farm management strategies that balance crop productivity with environmental impact, particularly N2O emissions. By modeling agricultural decision-making as a partially observable Markov decision process (POMDP), the framework accounts for uncertainties in environmental conditions and observational data. The approach integrates deep Q-learning with recurrent neural networks (RNNs) to train adaptive agents within a simulated farming environment. A Probabilistic Deep Learning (PDL) model was developed to estimate N2O emissions, achieving a high Prediction Interval Coverage Probability (PICP) of 0.937 within a 95% confidence interval on the available dataset. While the PDL model’s generalizability is currently constrained by the limited observational data, the RL framework itself is designed for broad applicability, capable of extending to diverse agricultural practices and environmental conditions. Results demonstrate that RL agents reduce N2O emissions without compromising yields, even under climatic variability. The framework’s flexibility allows for future integration of expanded datasets or alternative emission models, ensuring scalability as more field data becomes available. This work highlights the potential of artificial intelligence to advance climate-smart agriculture by simultaneously addressing productivity and sustainability goals in dynamic real-world settings.

The demand response (DR) program is a promising way to increase the ability to balance both supply and demand, optimizing the economic efficiency of the overall system. This study focuses on the DR participation strategy in terms of aggregators who offer appropriate DR programs to customers with flexible loads. DR aggregators engage in the electricity market according to customer behavior and must make decisions that increase the profits of both DR aggregators and customers. Customers use the DR program model, which sends its demand reduction capabilities to a DR aggregator that bids aggregate demand reduction to the electricity market. DR aggregators not only determine the optimal rate of incentives to present to the customers but can also serve customers and formulate an optimal energy storage system (ESS) operation to reduce their demands. This study formalized the problem as a Markov decision process (MDP) and used the reinforcement learning (RL) framework. In the RL framework, the DR aggregator and each customer are allocated to each agent, and the agents interact with the environment and are trained to make an optimal decision. The proposed method was validated using actual industrial and commercial customer demand profiles and market price profiles in South Korea. Simulation results demonstrated that the proposed method could optimize decisions from the perspective of the DR aggregator.

Relexi is an open source reinforcement learning (RL) framework written in Python and based on TensorFlow’s RL library TF-Agents. Relexi allows to employ RL for environments that require computationally intensive simulations like applications in computational fluid dynamics. For this, Relexi couples legacy simulation codes with the RL library TF-Agents at scale on modern high-performance computing (HPC) hardware using the SmartSim library. Relexi thus provides an easy way to explore the potential of RL for HPC applications.

The Internet of Things is shaping the next generation of cyber–physical systems to improve the future industry for smart cities. It has created novel and essential applications that require specific network performance to enhance the quality of services. Since network performance requirements are application-oriented, it is of paramount importance to provide tailored solutions that seamlessly manage the network resources and orchestrate the network to satisfy user requirements. In this article, we propose ELISE, a reinforcement learning (RL) framework to optimize the slotframe size of the time slotted channel hopping protocol in IIoT networks while considering the user requirements. We primarily address the problem of designing a framework that self-adapts to the optimal slotframe length that best suits the user's requirements. The framework takes care of all functionalities involved in the correct functioning of the network, while the RL agent instructs the framework with a set of actions to determine the optimal slotframe size each time the user requirements change. We evaluate the performance of ELISE through extensive analysis based on simulations and experimental evaluations on a testbed to demonstrate the efficiency of the proposed approach in adapting network resources at runtime to satisfy user requirements.

A Multi-Agent Reinforcement Learning Framework for Public Health Decision Analysis.

The increasing number of electric vehicles (EVs) necessitates the installation of more charging stations. The challenge of managing these grid-connected charging stations leads to a multi-objective optimal control problem where station profitability, user preferences, grid requirements and stability should be optimized. However, it is challenging to determine the optimal charging/discharging EV schedule, since the controller should exploit fluctuations in the electricity prices, available renewable resources and available stored energy of other vehicles and cope with the uncertainty of EV arrival/departure scheduling. In addition, the growing number of connected vehicles results in a complex state and action vectors, making it difficult for centralized and single-agent controllers to handle the problem. In this paper, we propose a novel Multi-Agent and distributed Reinforcement Learning (MARL) framework that tackles the challenges mentioned above, producing controllers that achieve high performance levels under diverse conditions. In the proposed distributed framework, each charging spot makes its own charging/discharging decisions toward a cumulative cost reduction without sharing any type of private information, such as the arrival/departure time of a vehicle and its state of charge, addressing the problem of cost minimization and user satisfaction. The framework significantly improves the scalability and sample efficiency of the underlying Deep Deterministic Policy Gradient (DDPG) algorithm. Extensive numerical studies and simulations demonstrate the efficacy of the proposed approach compared with Rule-Based Controllers (RBCs) and well-established, state-of-the-art centralized RL (Reinforcement Learning) algorithms, offering performance improvements of up to 25% and 20% in reducing the energy cost and increasing user satisfaction, respectively.

This article investigates the multirobot cooperative navigation problem based on raw visual observations. A fully end-to-end learning framework is presented, which leverages graph neural networks to learn local motion coordination and utilizes deep reinforcement learning to generate visuomotor policy that enables each robot to move to its goal without the need of environment map and global positioning information. Experimental results show that, with a few tens of robots, our approach achieves comparable performance with the state-of-the-art imitation learning-based approaches with bird-view state inputs. We also illustrate our generalizability to crowded and large environments and our scalability to ten times number of the training robots. In addition, we demonstrate that our model trained for multirobot case can also improve the success rate in the single-robot navigation task in unseen environments. Note to Practitioners—With the development of intelligent industrial and logistic systems, robotic transportation systems are widely implemented. However, existing multirobot path coordination and navigation approaches are basically under some unreasonable assumptions, which are very hard to be implemented in practical scenarios. This article aims to greatly promote the real application of learning-based multirobot cooperative navigation approach, in order to achieve the following. First, we introduce an end-to-end reinforcement learning framework instead of the commonly used imitation learning strategy, as the latter one needs exhaustive training data to cover all the scenarios and does not have the required generalizability. Second, we directly use the raw sensor data instead of the commonly used bird-eye-view semantic observations, as the latter one is generally not representative of practical application scenario from the robot perspective and cannot solve the occlusion issue. Third, we interpret our learned model to illustrate which parts of the input and shared observations contribute most to the robots’ final actions. The above interpretability ensures predictability (thus safety) of our visuomotor policy in practical applications. Our learned visuomotor policy has the ability to coordinate dozens of robots by only using raw visual observations in unknown environments without map nor global localization information, this is the first time in the literature. Our future work includes solving the sim-to-real issue and conducting physical experiments.

In this paper, the integration of a power system simulator and reinforcement learning (RL) tools and frameworks is presented. A proposed framework is easily applicable and can serve as a framework for further developing, training, and benchmarking RL algorithms on more complex tasks of power system control. The usage of standard RL frameworks enables a broad range of state-of-the-art algorithms to be implemented with high performance, scalability, and substantial code reuse. Also, the proposed framework design is suitable for scaling onto high-performance computing (HPC) clusters which significantly speeds up the computation. The IEEE 14-bus system is selected to show the simulation results of the proposed method.

“To do or not to do” is a fundamental decision that has to be made in daily life. Behaviors related to multiple “to do” choice tasks have long been explained by reinforcement learning, and “to do or not to do” tasks such as the go/no-go task have also been recently discussed within the framework of reinforcement learning. In this learning framework, alternative actions and/or the non-action to take are determined by evaluating explicitly given (overt) reward and punishment. However, we assume that there are real life cases in which an action/non-action is repeated, even though there is no obvious reward or punishment, because implicitly given outcomes such as saving physical energy and regret (we refer to this as “covert reward”) can affect the decision-making. In the current task, mice chose to pull a lever or not according to two tone cues assigned with different water reward probabilities (70% and 30% in condition 1, and 30% and 10% in condition 2). As the mice learned, the probability that they would choose to pull the lever decreased (<0.25) in trials with a 30% reward probability cue (30% cue) in condition 1, and in trials with a 10% cue in condition 2, but increased (>0.8) in trials with a 70% cue in condition 1 and a 30% cue in condition 2, even though a non-pull was followed by neither an overt reward nor avoidance of overt punishment in any trial. This behavioral tendency was not well explained by a combination of commonly used Q-learning models, which take only the action choice with an overt reward outcome into account. Instead, we found that the non-action preference of the mice was best explained by Q-learning models, which regarded the non-action as the other choice, and updated non-action values with a covert reward. We propose that “doing nothing” can be actively chosen as an alternative to “doing something,” and that a covert reward could serve as a reinforcer of “doing nothing.”

Financial institutions are increasingly dealing with difficulties in preserving transaction integrity due to escalating digital fraud and extensive cloud-based operations. Conventional machine learning and rule-based detection systems frequently have latency in replies, restricted adaptability, and an absence of verifiable audit trails. The proposed study presents a Hybrid Multi-Agent Reinforcement Learning (MARL) and Blockchain framework to overcome existing limitations, integrating adaptive multi-agent coordination for real-time fraud detection with a permissioned blockchain ledger to guarantee secure, tamper-proof validation of agent decisions. MARL agents adaptively acquire optimal detection strategies through incentive shaping and inter-agent communication, while blockchain consensus and smart contracts substantiate decision proofs for regulatory adherence. The experimental evaluation of the Kaggle Financial Transactions dataset demonstrates improved accuracy relative to baseline models: Accuracy 0.989, Precision 0.981, Recall 0.987, F1-Score 0.984, AUC-ROC 0.992, with negligible latency (1.8 ms/transaction). The comparative research reveals a persistent superiority over Random Forest, XGBoost, CNN- LSTM, and Transformer-based classifiers, showcasing robustness, scalability, and real-time application. The suggested study enhances the area by merging reinforcement learning adaptability with blockchain security, establishing a realistic, high-throughput, and reliable framework for cloud- based financial systems, while presenting a model for future research in agentic AI-driven fraud prevention.

Non-adaptive fuzzy logic controllers can become adaptive by learning from experience in the framework of reinforcement learning. In this paper, we discuss fuzzy reinforcement learning as a hybrid approach which provides a unified framework for including two types of prior knowledge: knowledge for control action selection and knowledge for performance evaluation. We describe GARIC, an architecture for combining fuzzy logic control and reinforcement learning, and apply it to cart-pole balancing and the Space Shuttle attitude control.

We have expected that dynamic higher functions such as “thinking” emerge through the growth from exploration in the framework of reinforcement learning (RL) using a chaotic Neural Network (NN). In this frame, the chaotic internal dynamics is used for exploration and that eliminates the necessity of giving external exploration noises. A special RL method for this framework has been proposed in which “traces” were introduced. On the other hand, reservoir computing has shown its excellent ability in learning dynamic patterns. Hoerzer et al. showed that the learning can be done by giving rewards and exploration noises instead of explicit teacher signals. In this paper, aiming to introduce the learning ability into our new RL framework, it was shown that the memory-required task in the work of Hoerzer et al. could be learned without giving exploration noises by utilizing the chaotic internal dynamics while the exploration level was adjusted flexibly and autonomously. The task could be learned also using “traces”, but still with problems.