Utility-Based Reinforcement Learning for Reactive Grids

Traffic signal control can be naturally regarded as a reinforcement learning problem. Unfortunately, it is one of the most difficult classes of reinforcement learning problems owing to its large state space. A straightforward approach to address this challenge is to control traffic signals based on continuous reinforcement learning. Although they have been successful in traffic signal control, they may become unstable and fail to converge to near-optimal solutions. We develop adaptive traffic signal controllers based on continuous residual reinforcement learning (CRL-TSC) that is more stable. The effect of three feature functions is empirically investigated in a microscopic traffic simulation. Furthermore, the effects of departing streets, more actions, and the use of the spatial distribution of the vehicles on the performance of CRL-TSCs are assessed. The results show that the best setup of the CRL-TSC leads to saving average travel time by 15% in comparison to an optimized fixed-time controller.

The paper is targeted at concerning pricing approach for grid resource scheduling with QoS guarantees. The two major parties in Grid, namely, resource consumers who submit various applications, and resources providers who share their resources, usually have different motivations when they join the Grid. These incentives are presented by objective functions in scheduling. Economic grid QoS scheduling algorithm adopt both application-centric and resource-centric scheduling objective function aim to optimize the performance of each individual grid user and the performance of the grid resources. The economic cost and profit are considered by grid users and resource providers respectively. Objective functions and QoS scheduling optimization algorithms are proposed. QoS scheduling optimization is decomposed into two sub optimizations of grid user's cost and grid resource provider's profit by utilizing dynamic programming. QoS scheduling algorithm is an interactive process in economic market coordinating grid users and grid resource providers to achieve optimal QoS scheduling. Experiments are to compare the proposed economic QoS scheduling with three partial optimisation approaches, which only consider one criterion.

A computational grid ensures the on-demand delivery of computing resources, in a security-aware, shared, scalable, and standards-based computing environment. A major concern is how to evolve a general and an encompassing framework that guarantees users' satisfaction measured as Quality of Services (QoS). To obtain a higher QoS, effective QoS perceived by subscribers (users) must conform to specified QoS agreements in the Service Level Agreements (SLAs) document--a legal contract between the Grid Services Provider (GSP) and users. Sometimes the effective user QoS does not conform to the specifications in the SLA because of the vagueness in linguistic definitions in the SLA. Existing approaches overcommitted resources to meet QoS. In this paper, we propose a fuzzy logic framework for calibrating a grid resources user-QoS that addresses the vagueness in linguistic definitions of the SLA document without overcommitting grid resources.

As data centers become essential large-scale infrastructures for data processing and intelligent computing, the efficiency of their internal scheduling systems is critical for both service quality and energy consumption. The performance of these scheduling systems significantly impacts the quality of computing services and overall energy usage. However, the rapid increase in task volume, coupled with the diversity of computing resources, poses substantial challenges to traditional scheduling approaches. Conventional container scheduling approaches typically focus on either minimizing task execution time or reducing energy consumption independently, often neglecting the importance of balancing these two objectives simultaneously. In this study, a container scheduling algorithm based on the Soft Actor–Critic framework, called SAC-CS, is proposed. This algorithm aims to enhance container execution efficiency while concurrently reducing energy consumption in data centers. It employs a maximum entropy reinforcement learning approach, enabling a flexible trade-off between energy use and task completion times. Experimental evaluations on both synthetic workloads and Alibaba cluster datasets demonstrate that the SAC-CS algorithm effectively achieves joint optimization of efficiency and energy consumption, outperforming heuristic methods and alternative reinforcement learning techniques.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

Within the cloud manufacturing paradigm, Cloud Manufacturing Service Composition (CMSC) is a core technology for intelligent resource orchestration in Cloud Manufacturing Platforms (CMP). However, existing research faces critical limitations in real-world CMP operations: single-task-centric optimization ignores resource sharing/competition among coexisting manufacturing tasks (MTs), causing performance degradation and resource “starvation”; traditional heuristics require full re-execution for new scenarios, failing to support real-time online decision-making; single-agent reinforcement learning (RL) lacks mechanisms to balance global efficiency and inter-task fairness, suffering from inherent fairness defects. To address these challenges, this paper proposes a fairness-aware multi-task CMSC method based on Multi-Agent Reinforcement Learning (MARL) under the Centralized Training with Decentralized Execution (CTDE) framework, targeting the symmetry-breaking issue of uneven resource allocation among MTs and aiming to achieve symmetry restoration by restoring relative balance in resource acquisition. The method constructs a multi-task CMSC model that captures real-world resource sharing/competition among concurrent MTs, and integrates a centralized global coordination agent into the MARL framework (with independent task agents per MT) to dynamically regulate resource selection probabilities, overcoming single-agent fairness defects while preserving distributed autonomy. Additionally, a two-layer attention mechanism is introduced—task-level self-attention for intra-task subtask correlations and global state self-attention for critical resource features—enabling precise synergy between local task characteristics and global resource states. Experiments verify that the proposed method significantly enhances inter-task fairness while maintaining superior global Quality of Service (QoS), demonstrating its effectiveness in balancing efficiency and fairness for dynamic multi-task CMSC.

AQROM: A quality of service aware routing optimization mechanism based on asynchronous advantage actor-critic in software-defined networks

The aim of this thesis is to design and implement a new Grid Resource Management methodology, where non-massive resources owners would be able to share their resources and integrate human collaboration across multiple domains regardless of network technology, operative platform or administrative domain.<br/>This thesis proposes a distributed and heuristic policy-based resource management architecture for large-scale Grids. The resource management architecture proposed herein is composed of four main building blocs: services management, resource discovery and monitoring, resource scheduling and jobs allocation and activation. The Grid Services Management (GSM) and Jobs Allocation and Activation (JAA) are supported by means of a Policy-based Grid Resource Management Architecture (PbGRMA). This architecture is able to identify service needs arising from diverse sources during the deployment and management of Grid Services, such as requirements demanded by customers, applications and network conditions. Afterwards, the PbGRMA merges these requirements into deployment policies for the corresponding Grid Services. The Grid Resource Discovery and Monitoring (GRDM) is supported by the introduction of the SNMP-based Balanced Load Monitoring Agents for Resource Scheduling (SBLOMARS), in which network and computational resources are monitored by distributed agents. This allows for a flexible, heterogeneous and scalable monitoring system. The Grid Resource Scheduling (GRS) is based on the Balanced Load Multi-Constrained Resource Scheduler (BLOMERS). This heuristic scheduler represents an alternate way of solving the inherent NP-hard problem for resource scheduling in large-scale distributed networks by means of the implementation of a Genetic Algorithm.<br/>Finally, based on the outcome of both the GRDM and GRS, the PbGRMA allocates the corresponding Grid Services by means of its interfaces with Globus ToolKit Middleware and Unix-based CLI commands along of any large-scale Grid Infrastructure. <br/>The synergy obtained by these components allows Grid administrators to exploit the available resources with predetermined levels of Quality of Service (QoS), reducing computational costs and makespan in resource scheduling while ensuring that the resource load is balanced throughout the Grid. The makespan of a schedule is the time required for all jobs to be processed when no one job could be interrupted during its execution and each node can perform at most one operation at any time.<br/>This new approach has been successfully tested in a real large-scale scenario such as Grid5000. The results presented along this Thesis show that our general solution is a reliable, flexible and scalable architecture to deploy and manage Grid Services in large-scale Grid Infrastructures. Moreover, the substitution of the heuristic algorithm approach used into the Grid Resource Scheduling (GRS) phase by other non-heuristics selection algorithms could make our solution useful in smaller Grid Infrastructures.

In continual learning, an agent is exposed to a changing environment, requiring it to adapt during execution time. While traditional reinforcement learning (RL) methods have shown impressive results in various domains, there has been less progress in addressing the challenge of continual learning. Current RL approaches do not allow the agent to adapt during execution but only during a dedicated training phase. Here we study the problem of continual learning in a 2D bipedal walker domain, in which the legs of the walker grow over its lifetime, requiring the agent to adapt. The introduced approach combines neuroevolution, to determine the starting weights of a deep neural network, and a version of deep reinforcement learning that is continually running during execution time. The proof-of-concept results show that the combined approach gives a better generalisation performance when compared to evolution or reinforcement learning alone. The hybridization of reinforcement learning and evolution opens up exciting new research directions for continually learning agents that can benefit from suitable priors determined by an evolutionary process.

The Grid can be seen as a collection of services each of which performs some functionality. Grid users often submit their applications in the form of workflows with certain Quality of Service (QoS) requirements imposed on the workflows. These workflows detail the composition of Grid services and the level of service required from the Grid. This paper addresses workload allocation techniques for Grid workflows. We model a Grid service as a G/G/1 queue and minimise failures (QoS requirement violation) of jobs by solving a mixed-integer non-linear program (MINLP). The novel approach is evaluated through an experimental simulation and the results confirm that the proposed workload allocation strategy performs considerably better in terms of satisfying QoS requirements of Grid workflows than scheduling algorithms that don?t employ such workload allocation techniques.

Research about air-craft autonomous vehicles has become a trend issue among researchers. Vertical Take-Off and Landing (VTOL) controlling for a hover-craft vehicle is one of benchmarking control problems. VTOL ability of hover-craft is significantly crucial for the vehicle to do surveillance and rescue, military, farming, etc. There are many papers concerned with this VTOL benchmarking control. The purpose of this paper is applying Deep Deterministic Policy Gradient (DDPG) as the brand new continuous deep reinforcement learning approach to solve a complex nonlinear system, such as air-craft VTOL problem. Furthermore, this paper analyses the effect of Ornstein-Uhlenbeck (OU) Noise injection to action as the exploration key issue of DDPG algorithm, in addition of the analysis, this paper also applies the gaussian noise to the plant as the disturbance of system. The test was performed in the VTOL simulation. The evaluation used the episode reward, average reward, total reward from the training process. Further analysis also uses IAE (Integral of Absolute Magnitude of Error), ISE (Integral of the Square Error) and MSE (Mean of the Square Error) as the parameter error evaluation from the system. The comparison results of all conditions this paper show that applying DDPG with the injection of OU noise to VTOL system with the disturbance of Gaussian noise achieves the best total reward of −14,355.37 with the shortest episode training of 243, thus it works best. The VTOL system without the disturbance has higher IAE and VTOL with Gaussian noise as the disturbance has higher ISE and MSE.

Focuses on the development of a learning-based heuristic for scheduling heterogeneous machines. Although list scheduling methods have been widely used for a large class of scheduling problems, including the heterogeneous machine scheduling problem, they involve designing priority rules, which usually require a fair amount of insights on the characteristics of the problem to be solved. Instead of elaborate design of priority rules in a single step, we propose an iterative list scheduling process, which refines priority rules while generating a number of schedules. The proposed iterative list scheduling is formulated as a reinforcement learning problem, with states and actions defined in list scheduling. Due to the large number of possible states, reinforcement learning algorithms which use value functions in constructing an optimal policy may not be suitable for scheduling problems. Thus, to directly work with policies rather than the values of states, we propose genetic reinforcement learning (GRL), in which the policies of reinforcement learning are encoded into the chromosomes of genetic algorithms and a near-optimal policy is searched for by genetic algorithms. A GRL-based scheduler, called evolutionary intracell scheduler (EVIS), has been developed and applied to various scheduling problems such as the heterogeneous machine scheduling, the processor scheduling, the job-shop scheduling, the flow-shop scheduling, and the open-shop scheduling problems. The proposed model of EVIS, which has a linear order of population-fitness convergence, is verified by computer experiments. Even without fine tuning EVIS, the quality of solutions achieved by EVIS is comparable to that of problem-tailored heuristics for most of the problem instances.

Ad hoc grids allow a group of individuals to accomplish a mission that involves computation and communication among the grid components, often without fixed structure. In an ad hoc grid, every node in the network can spontaneously arise as a resource consumer or a resource producer at any time when it needs a resource or it possesses an idle resource. At the same time, the node in ad hoc grid is often energy constrained. The paper proposes an efficient resource allocation scheme for grid computing marketplace where ad hoc grid users can buy usage of memory and CPU from grid resource providers. The ad hoc grid user agents purpose to obtain the optimized quality of service to accomplish their tasks on time with a given budget, and the goal of grid resource providers as profit-maximization. Combining perspectives of both ad hoc grid users and resource providers, the paper present ad hoc grid resource allocation algorithm to maximize the global utility of the ad hoc grid system which are beneficial for both grid users and grid resource providers. Simulations are conducted to compare the performance of the algorithms with related work.

Inventory management remains a cornerstone of effective supply chain performance, directly influencing cost efficiency, service quality, and organizational agility. In today’s hypercompetitive and uncertain market environment, inventory decisions must account for complex variables such as fluctuating demand, supply disruptions, lead time variability, and market seasonality. Traditional inventory control models such as the Economic Order Quantity (EOQ), base-stock policies, and (s, S) strategies are often static in nature. They rely on pre-defined parameters and assume stationarity in demand and supply, limiting their ability to respond dynamically to real-time changes. In contrast, Reinforcement Learning (RL) offers a paradigm shift in how inventory decisions can be optimized. As a subfield of machine learning, RL enables agents to learn optimal strategies through repeated interactions with an environment, using trial-and-error exploration and reward-based feedback. RL agents can observe the system state (e.g., inventory levels, demand signals, lead time status), choose actions (e.g., place an order or wait), and receive feedback in the form of rewards (e.g., service level achievements or cost penalties), thus iteratively improving their policies. This study explores how RL can be applied to optimize inventory management in environments characterized by uncertainty and real-time decision-making needs. Specifically, we investigate how different RL algorithms such as Q-learning, Deep Q Networks (DQN), and Policy Gradient methods perform in various inventory scenarios. Additionally, we examine the computational and operational implications of deploying RL in real-world settings, including issues of model convergence, exploration-exploitation tradeoffs, data requirements, and scalability. We also discuss how RL can complement other AI techniques such as demand forecasting models and predictive analytics in creating end-to-end intelligent supply chain solutions. By bridging the gap between theoretical RL frameworks and practical inventory management applications, this paper contributes to both the academic literature and industrial practice. Our goal is to demonstrate that RL is not only a theoretically elegant solution but also a viable tool for achieving inventory efficiency and supply chain resilience.

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.