Efficient Resource Management for Real-time AI Systems in the Cloud using Reinforcement Learning

Reinforcement Learning (RL) is a sampling based approach to optimization, where learning agents rely on scalar reward signals to discover optimal solutions. The specification of learning objectives as scalar rewards is tedious and error prone, and more so for real-time systems with complex time-critical requirements. This paper advocates the use of Duration Calculus (DC)—a highly expressive real-time logic with duration and length modalities—in expressing the learning objectives in model-free RL for stochastic real-time systems. On the other hand, to model stochastic real-time environments, we consider probabilistic timed automata (PTA)—Markov decision processes extended with clock variables—that provide an expressive yet computationally decidable formalism to capture real-time constraints over nondeterministic and probabilistic behaviors.The key hurdle in developing a convergent RL algorithm for DC specifications is the undecidability of the synthesis problem for PTA against general DC specifications. Inspired by the dichotomy between event-triggered and time-triggered approaches to the design of real-time systems, we present two variants of DC logic—that we dub event-triggered duration calculus (EDC) and time-triggered duration calculus (TDC)—and identify their subclasses with appealing theoretical properties. We study the decidability (and exact complexity) of the satisfiability of these calculi as well as the controller synthesis against PTA models. Based on these results, we propose a reward scheme for RL agents in such a way that guarantees that any RL algorithm maximizing rewards is guaranteed to maximize the probability of satisfaction for the given DC specification. The effectiveness of the proposed approach is demonstrated via grid-world benchmarks and a proof-of-concept case study for synthesizing control for simple cardiac pacemaker directly from a set of DC specifications.

Trajectory smoothing method using reinforcement learning for computer numerical control machine tools

Relational reinforcement learning is a technique that combines reinforcement learning with relational learning or inductive logic programming. This technique offers greater expressive power than that one offered by traditional reinforcement learning. However, there are some problems when one wish to use it in a real time system. Most of recent research interests on incremental relational learning structure, that is a great challenge in this area. In this work, we are proposing an enhancement of TG algorithm and we illustrate the approach with a preliminary experiment. The algorithm was evaluated on a Blocks World simulator and the obtained results shown it is able to produce appropriate learn capability.

6G, the next generation of mobile networks, is set to offer even higher data rates, ultra-reliability, and lower latency than 5G. New 6G services will increase the load and dynamism of the network. Network Function Virtualization (NFV) aids with this increased load and dynamism by eliminating hardware dependency. It aims to boost the flexibility and scalability of network deployment services by separating network functions from their specific proprietary forms so that they can run as virtual network functions (VNFs) on commodity hardware. It is essential to design an NFV orchestration and management framework to support these services. However, deploying bulky monolithic VNFs on the network is difficult, especially when underlying resources are scarce, resulting in ineffective resource management. To address this, microservices-based NFV approaches are proposed. In this approach, monolithic VNFs are decomposed into 'micro' VNFs, increasing the likelihood of their successful placement and resulting in more efficient resource management. This article discusses the proposed framework for resource allocation for microservices-based services to provide end-to-end Quality of Service (QoS) using the Double Deep Q Learning (DDQL) approach. Furthermore, to enhance this resource allocation approach, we discussed and addressed two crucial sub-problems: the need for a dynamic priority technique and the presence of the low-priority starvation problem. Using the Deep Deterministic Policy Gradient (DDPG) model, an Adaptive Scheduling model is developed that effectively mitigates the starvation problem. Additionally, the impact of incorporating traffic load considerations into deployment and scheduling is thoroughly investigated.

Deep reinforcement learning (RL) is achieving significant success in various applications like control, robotics, games, resource management, and scheduling. However, the important problem of emergency evacuation, which clearly could benefit from RL, has been largely unaddressed. Indeed, emergency evacuation is a complex task that is difficult to solve with RL. An emergency situation is highly dynamic, with a lot of changing variables and complex constraints that make it challenging to solve. Also, there is no standard benchmark environment available that can be used to train RL agents for evacuation. A realistic environment can be complex to design. In this article, we propose the first fire evacuation environment to train RL agents for evacuation planning. The environment is modeled as a graph capturing the building structure. It consists of realistic features like fire spread, uncertainty, and bottlenecks. The implementation of our environment is in the OpenAI gym format, to facilitate future research. We also propose a new RL approach that entails pretraining the network weights of a DQN-based agent [DQN/Double-DQN (DDQN)/Dueling-DQN] to incorporate information on the shortest path to the exit. We achieved this by using tabular <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning to learn the shortest path on the building model’s graph. This information is transferred to the network by deliberately overfitting it on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -matrix. Then, the pretrained DQN model is trained on the fire evacuation environment to generate the optimal evacuation path under time varying conditions due to fire spread, bottlenecks, and uncertainty. We perform comparisons of the proposed approach with state-of-the-art RL algorithms like DQN, DDQN, Dueling-DQN, PPO, VPG, state-action-reward-state-action (SARSA), actor–critic method, and ACKTR. The results show that our method is able to outperform state-of-the-art models by a huge margin including the original DQN-based models. Finally, our model is tested on a large and complex real building consisting of 91 rooms, with the possibility to move to any other room, hence giving 8281 actions. In order to reduce the action space, we propose a strategy that involves one step simulation. That is, an action importance vector is added to the final output of the pretrained DQN and acts like an attention mechanism. Using this strategy, the action space is reduced by 90.1%. In this manner, the model is able to deal with large action spaces. Hence, our model achieves near optimal performance on the real world emergency environment.

A bio-inspired neuro-adaptive deep reinforcement learning approach for real-time solar tracking system to enhance photovoltaic efficiency

The process industry (PI) plays a vital role in the global economy and faces mounting pressure to enhance sustainability, operational agility, and resource efficiency amid tightening regulatory and market demands. Although artificial intelligence (AI) has been explored in this domain for decades, its adoption in industrial practice remains limited. Recently, machine learning (ML) has gained momentum, particularly when integrated with core PI systems such as process control, instrumentation, quality management, and enterprise platforms. Among ML techniques, reinforcement learning (RL) has emerged as a promising approach to tackle complex operational challenges. In contrast to conventional data-driven methods that focus on prediction or classification, RL directly addresses sequential decision making under uncertainty, a defining characteristic of dynamic process operations. Given RL’s growing relevance, this study conducts a systematic literature review to evaluate its current applications in the PI, assess methodological developments, and identify barriers to broader industrial adoption. The review follows the PRISMA methodology, a structured framework for identifying, screening, and selecting relevant publications. This approach ensures alignment with a clearly defined research question and minimizes bias, focusing on studies that demonstrate meaningful industrial applications of RL. The findings reveal that RL is transitioning from a theoretical construct to a practical tool, particularly in the chemical sector and for tasks such as process control and scheduling. Methodological maturity is improving, with algorithm selection increasingly tailored to problem-specific requirements and a trend toward hybrid models that integrate RL with established control strategies. However, most implementations remain confined to simulated environments, underscoring the need for real-world deployment, safety assurances, and improved interpretability. Overall, RL exhibits the potential to serve as a foundational component of next-generation smart manufacturing systems.

Ensuring that AI systems do what we, as humans, actually want them to do is one of the biggest open research challenges in AI alignment and safety. My research seeks to directly address this challenge by enabling AI systems to interact with humans to learn aligned and robust behaviors. The way robots and other AI systems behave is often the result of optimizing a reward function. However, manually designing good reward functions is highly challenging and error‐prone, even for domain experts. Although reward functions are often difficult to manually specify, human feedback in the form of demonstrations or preferences is often much easier to obtain but can be difficult to interpret due to ambiguity and noise. Thus, it is critical that AI systems take into account epistemic uncertainty over the human's true intent. As part of the AAAI New Faculty Highlight Program, I will give an overview of my research progress along the following fundamental research areas: (1) efficiently quantifying uncertainty over human intent, (2) directly optimizing behavior to be robust to uncertainty over human intent, and (3) actively querying for additional human input to reduce uncertainty over human intent.

Various Internet solutions take their power processing and analysis from cloud computing services. Internet of Things (IoT) applications started discovering the benefits of computing, processing, and analysis on the device itself aiming to reduce latency for time-critical applications. However, on-device processing is not suitable for resource-constraints IoT devices. Edge computing (EC) came as an alternative solution that tends to move services and computation more closer to consumers, at the edge. In this letter, we study and discuss the applicability of merging deep learning (DL) models, i.e., convolutional neural network (CNN), recurrent neural network (RNN), and reinforcement learning (RL), with IoT and information-centric networking which is a promising future Internet architecture, combined all together with the EC concept. Therefore, a CNN model can be used in the IoT area to exploit reliably data from a complex environment. Moreover, RL and RNN have been recently integrated into IoT, which can be used to take the multi-modality of data in real-time applications into account.

To support the 6G vision of seamless “space–air–ground-integrated” global coverage, optical satellite networks must enable high-speed, low-latency, and intelligent data transmission. However, conventional inter-satellite laser link-based optical transport networks suffer from inefficient bandwidth utilization and nonlinear latency accumulation caused by multi-hop routing, which severely limits their ability to support ultra-low-latency and real-time applications. To address the critical challenges of high topological complexity and stringent real-time requirements in satellite elastic optical networks, we propose an asynchronous advantage actor–critic-based quality of service-aware routing optimization mechanism for the optical inter-satellite link (OISL-AQROM). By establishing a quantitative model that correlates the optical service unit (OSU) C value with node hop count, the algorithm enhances the performance of latency-sensitive services in dynamic satellite environments. Simulation results conducted on a Walker-type low Earth orbit (LEO) constellation comprising 1152 satellites demonstrate that OISL-AQROM reduces end-to-end latency by 76.3% to 37.6% compared to the traditional heuristic multi-constrained shortest path first (MCSPF) algorithm, while supporting fine-grained dynamic bandwidth adjustment down to a minimum granularity of 2.6 Mbps. Furthermore, OISL-AQROM exhibits strong convergence and robust stability across diverse traffic loads, consistently outperforming MCSPF and deep deterministic policy gradient (DDPG) algorithm in overall efficiency, load adaptability, and operational reliability. The proposed algorithm significantly improves service quality and transmission efficiency in commercial mega-constellation optical satellite networks, demonstrating engineering applicability and potential for practical deployment in future 6G infrastructure.

The growth of the Internet of Things (IoT) and smart systems has paved the way for the integration of machine learning (ML) and optimization techniques to enhance their efficiency, security, and scalability. This survey provides a comprehensive overview of recent advancements in applying machine learning algorithms and optimization strategies to IoT and smart systems. It highlights key areas where these technologies intersect, such as energy efficiency, intelligent decision-making, resource management, and anomaly detection. The paper discusses various machine learning approaches, including deep learning, reinforcement learning, and supervised/unsupervised models, that have been applied to optimize system performance in real-time applications. Furthermore, the survey examines optimization techniques like metaheuristics, evolutionary algorithms, and gradient-based methods that improve IoT systems' reliability and resource utilization. The integration of these techniques offers significant advantages, including reduced latency, enhanced security, and increased adaptability in smart systems. The findings emphasize the importance of adopting ML and optimization methods to address the challenges posed by the growing complexity of IoT environments, fostering future innovations in connected and autonomous systems. This survey serves as a resource for researchers and practitioners seeking to explore the synergies between IoT, ML, and optimization techniques.

Rapid expansion of the Internet of Things (IoT) and mobile applications has increased the demand for low latency and efficient computational resource management. Although offering high processing power, cloud computing suffers from network congestion and latency issues, making edge computing a viable alternative. Task offloading in cloud-edge environments is crucial for optimizing resource allocation, reducing delays, and enhancing system performance. For intelligent job offloading in an Edge-Cloud architecture, this study suggests a hybrid reinforcement learning model based on Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO). The model is trained on the datasets simulated in EdgeCloudSim simulator and consisting of tasks, network conditions, computational capabilities of servers, and energy efficiency constraints. The results of the experiment shows that the suggested model considerably reduces task execution time, lowers energy usage, and boosts system efficiency. This work provides a robust deep reinforcement learning-based solution for optimizing task offloading in future edge-cloud computing infrastructures.

Efficient management of renewable energy resources is imperative for promoting environmental sustainability and optimizing the utilization of clean energy sources. This paper presents a pioneering European-scale study on energy management within renewable energy communities (RECs). With a primary focus on enhancing the social welfare of the community, we introduce a reinforcement learning (RL) controller designed to strategically manage Battery Energy Storage Systems (BESSs) and orchestrate energy flows. This research transcends geographical boundaries by conducting an extended analysis of various energy communities and diverse energy markets across Europe, encompassing different regions of Italy. Our methodology involves the implementation of an RL controller, leveraging optimal control theory for training and utilizing only real-time data available at the current time step during the test phase. Through simulations conducted in diverse contexts, we demonstrate the superior performance of our RL agent compared to a state-of-the-art rule-based controller. The agent exhibits remarkable adaptability to various scenarios, consistently surpassing existing rule-based controllers. Notably, we illustrate that our approach aligns with the intricate patterns observed in both Italian and European energy markets, achieving performance levels comparable to an optimal controller assuming perfect theoretical knowledge of future data.

This research work aims at evaluating the possibilities of using artificial intelligence in dynamic resource management and optimization of software system performance. In today’s complex world of application usage, normal methods of resource management are unable to cater to these dynamic needs and fulfill its usage potential. In this work, an assessment of three mainstream AI techniques – reinforcement learning, neural network, and genetic algorithm – is performed based on performance indicators such as resource utilization and consumption, average response time, throughput, costs, prediction capability, stability, and time taken to converge. The results show that the neural networks have the best resource acquisition performance as well as response rates, while the reinforcement learning has the best cost management and flexibility rates. As it has been pointed out, genetic algorithms are quite useful in finding optimization solutions, however real-time responsiveness is lack. Thus, the results provide significant understating of how to choose the proper AI technique depending on the specific application needs which in turn will be useful for organizations willing to improve their resource management using AI-based solutions.

Wireless Sensor Networks (WSNs) play a critical role in environmental monitoring, healthcare, disaster management, and smart infrastructure. However, the limited energy resources of sensor nodes remain a pressing challenge, particularly in data aggregation and transmission processes, where redundancy and inefficient routing can significantly shorten network lifetime. To address this problem, we propose a Hybrid Deep Reinforcement Learning (HDRL) framework that optimizes data aggregation while balancing energy consumption and communication overhead. The method integrates the decision making capability of reinforcement learning with the representational power of deep neural networks, enabling adaptive node selection and dynamic routing based on real-time energy and network states. The proposed HDRL model employs a dual-agent mechanism: the first agent focuses on cluster head selection for balanced energy distribution, while the second agent optimizes multi-hop routing paths to minimize redundant transmissions. A reward function is designed to jointly consider residual energy, data latency, and transmission reliability. Simulation results show that the HDRL-based approach outperforms traditional clustering and reinforcement learning methods in terms of network lifetime extension, reduced packet loss, and improved throughput. Notably, the proposed method achieves up to 30% improvement in energy efficiency and 25% reduction in end-to end delay, making it highly suitable for large-scale, real-time WSN applications.