A Knowledge Graph-based Cooperative Caching Scheme in MEC-enabled Heterogeneous Networks

In this paper, we propose a novel deep reinforcement learning (DRL) method for optimal path planning for mobile robots using dynamic programming (DP)-based data collection. The proposed method can overcome the slow learning process and improve training data quality inherently in DRL algorithms. The main idea of our approach is as follows. First, we mapped the dynamic programming method to typical optimal path planning problems for mobile robots, and created a new efficient DP-based method to find an exact, analytical, optimal solution for the path planning problem. Then, we used high-quality training data gathered using the DP method for DRL, which greatly improves training data quality and learning efficiency. Next, we established a two-stage reinforcement learning method where, prior to the DRL, we employed extreme learning machines (ELM) to initialize the parameters of actor and critic neural networks to a near-optimal solution in order to significantly improve the learning performance. Finally, we illustrated our method using some typical path planning tasks. The experimental results show that our DRL method can converge much easier and faster than other methods. The resulting action neural network is able to successfully guide robots from any start position in the environment to the goal position while following the optimal path and avoiding collision with obstacles.

The hybrid flow shop scheduling problem (HFSP) has been extensively studied due to its wide applications in modern manufacturing systems. However, existing research primarily focuses on machine-centric scheduling while neglecting the critical role of human resources, whose varying skill levels impact processing efficiency and whose availability constraints complicate scheduling decisions. Therefore, this paper investigates the HFSP with human resource constraints (HFSP-HRC), which requires the joint optimization of job sequencing, machine allocation, and worker assignment to obtain efficient scheduling solutions. We propose a deep reinforcement learning (DRL) method integrated with a meta-path-based heterogeneous graph neural network (MP-HGNN) to solve this problem. It innovatively captures complex interdependencies among operations, machines, and workers through machine-centric and worker-centric meta-paths, overcoming limitations of existing methods that fail to model heterogeneous resource competition relationships. The integration enables high-order semantic feature fusion for enhanced scheduling robustness. Combined with the proximal policy optimization (PPO) algorithm, the framework realizes end-to-end learning from raw scheduling states to optimal decisions. Experiments across four different problem scales demonstrate that the proposed method significantly outperforms baseline heuristic rules and DRL method in makespan, with particularly outstanding performance on large-scale instances.

In this article, a new deep reinforcement learning (RL) method, called asynchronous advantage actor-critic (A3C) method, is developed to solve the optimal control problem of elevator group control systems (EGCSs). The main contribution of this article is that the optimal control law of EGCSs is designed via a new deep RL method, such that the elevator system sends passengers to the desired destination floors as soon as possible. Deep convolutional and recurrent neural networks, which can update themselves during applications, are designed to dispatch elevators. Then, the structure of the A3C method is developed, and the training phase for the learning optimal law is discussed. Finally, simulation results illustrate that the developed method effectively reduces the average waiting time in a complex building environment. Comparisons with traditional algorithms further verify the effectiveness of the developed method.

The study examines the complex processes of intelligent management of energy resilience in modern healthcare facilities during critical situations, large-scale failures, and prolonged outages of the external centralized power supply. The aim is to develop a comprehensive mathematical model and a Safe Deep Reinforcement Learning (Safe DRL) method that ensures guaranteed compliance with the strict physical and operational constraints of hospital energy systems, even during the intensive training phase of a neural network agent. The objectives are: to formalize in detail the decision-making procedure in an energy system in detail by transitioning to the paradigm of Constrained Markov Decision Processes (CMDP); to develop an innovative mathematical model featuring the implementation of a specialized safety layer based on Lyapunov functions; and to ensure high resilience and autonomy of the system through the implementation of a decentralized Edge-Fog data processing architecture. The methods used include: the theory of Constrained Markov Decision Processes (CMDP), deep reinforcement learning methods based on the Actor-Critic architecture, the mathematical apparatus of Lyapunov stability theory for the analytical correction of actions, and methods of simulation modeling for complex dynamic energy systems. The following results were obtained. In the course of the study, a Safe DRL method was proposed and substantiated, which integrates a Lyapunov-based projection directly into the training loop for the immediate correction of the agent’s control actions. This makes it possible to ensure strict theoretical guarantees of maintaining the required State of Charge (SoC) of battery systems and to prevent critical violations of energy system parameters beyond established safety limits, which is crucial for patient life-support. The effectiveness of the proposed approach was confirmed by a series of numerical experiments in the specialized environment, HospitalEnergyEnv. Under a full blackout scenario, the agent demonstrated adaptability and high accuracy in resource management without any violation of the established physical limits during the entire process of autonomous operation. Conclusions. The scientific novelty of the obtained results lies in the following: the existing optimization model for building energy management systems (BEMS) has been improved by introducing an analytical safety projection mechanism, which minimizes the risks of emergency equipment shutdown during the adaptation of artificial intelligence algorithms; further development of decentralized control methods for critical infrastructure based on Edge-Fog computing has been achieved, which significantly increases system fault tolerance in the event of a loss of connection with the global network and ensures obtaining quasi-optimal solutions in high-dimensional problems. The practical value of this work lies in the potential to create highly reliable autonomous energy systems for critical infrastructure facilities.

Recently, the development of Low Earth Orbit (LEO) satellites and the advancement of the Mobile Edge Computing (MEC) paradigm have driven the emergence of the Satellite Mobile Edge Computing (Sat-MEC). Sat-MEC has been developed to support communication and task computation for Internet of Things (IoT) Mobile Devices (IMDs) in the absence of terrestrial networks. However, due to the heterogeneity of tasks and Sat-MEC servers, it is still a great challenge to efficiently schedule tasks in Sat-MEC servers. Here, we propose a scheduling algorithm based on the Deep Reinforcement Learning (DRL) method in the Sat-MEC architecture to minimize the average task processing time. We consider multiple factors, including the cooperation between LEO satellites, the concurrency and heterogeneity of tasks, the dynamics of LEO satellites, the heterogeneity of the computational capacity of Sat-MEC servers, and the heterogeneity of the initial queue for task computation. Further, we use the self-attention mechanism to act as a Q-network to extract high-dimensional dynamic information of tasks and Sat-MEC servers. In this work, we model the Sat-MEC environment simulation at the application level and propose a DRL-based task scheduling algorithm. The simulation results confirm the effectiveness of our proposed scheduling algorithm, which reduces the average task processing time by 22.1%, 30.6%, and 41.3%, compared to the genetic algorithm(GA), the greedy algorithm, and the random algorithm, respectively.

First-person shooter (FPS) video games play an important role in game artificial intelligence (AI). In this paper, we present an effective deep reinforcement learning (DRL) method to learn battles in ViZDoom. Our approach utilizes the actor-critic with Kronecker-factored trust region (ACKTR), a sample-efficient and computationally inexpensive DRL method. We train our ACKTR agents in two battle scenarios, and compare with the advantage actor-critic (A2C) baseline agent. The experimental results demonstrate that DRL methods successfully teach agents to battle in these scenarios. In addition, the ACKTR agents significantly outperform the A2C agents in terms of all the metrics by a significant margin.

Adaptive computation offloading and resource allocation strategy in a mobile edge computing environment

Iterative rule-guided reasoning over sparse knowledge graphs with deep reinforcement learning

In the manufacturing industry, the design of machining processes plays a pivotal role in determining the quality, efficiency, and cost of product manufacturing. Machining process design is progressing towards intelligence to meet the high demands for efficiency and effectiveness in smart manufacturing. Building upon traditional computer-aided process planning, intelligent technologies, such as machine learning and knowledge graphs, have emerged as key drivers in advancing intelligent process design. To address knowledge accumulation, inflexible reuse, and fragmented reasoning in machining process design, this paper organically integrates the structured representation characteristics of the knowledge graph and the perceptual reasoning capabilities of deep reinforcement learning. It introduces an intelligent generation method for machining process design based on knowledge graph and deep reinforcement learning, aiming to achieve unified representation and reasoning reuse for historical process cases and general process rules. The effectiveness of the proposed method is validated through a case study involving the generation of machining process solutions for a specific model of diesel engine components. This research contributes valuable insights to overcoming the limitations of traditional methods and enhancing the efficiency of the machining process design.

The Internet of Things (IoT) has been continuously rising in the past few years, and its potentials are now more apparent. However, transient data generation and limited energy resources are the major bottlenecks of these networks. Besides, minimum delay and other conventional quality of service measurements are still valid requirements to meet. An efficient caching policy can help meet the standard quality of service requirements while bypassing IoT networks' specific limitations. Adopting deep reinforcement learning (DRL) algorithms enables us to develop an effective caching scheme without needing prior knowledge or contextual information. In this work, we propose a DRL-based caching scheme that improves the cache hit rate and reduces energy consumption of the IoT networks, in the meanwhile, taking data freshness and limited lifetime of IoT data into account. To better capture the regional-different popularity distribution, we adopt a hierarchical architecture to deploy edge caching nodes in IoT networks. The results of comprehensive experiments show that our proposed method outperforms the well-known conventional caching policies and an existing DRL-based solution in terms of cache hit rate and energy consumption of the IoT networks by considerable margins.

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

With the continuous development of the 6G mobile network, computing-intensive and delay-sensitive onboard applications generate task data traffic more frequently. Particularly, when multiple intelligent agents are involved in tasks, limited computational resources cannot meet the new Quality of Service (QoS) requirements. To provide a satisfactory task offloading strategy, combining Multi-Access Edge Computing (MEC) with artificial intelligence has become a potential solution. In this context, we have proposed a task offloading decision mechanism (TODM) based on cooperative game and deep reinforcement learning (DRL). A joint optimization problem is presented to minimize both the overall task processing delay (OTPD) and overall task energy consumption (OTEC). The approach considers task vehicles (TaVs) and service vehicles (SeVs) as participants in a cooperative game, jointly devising offloading strategies to achieve resource optimization. Additionally, a proximate policy optimization (PPO) algorithm is designed to ensure robustness. Simulation experiments confirm the convergence of the proposed algorithm. Compared with benchmark algorithms, the presented scheme effectively reduces delay and energy consumption while ensuring task completion.

Automated driving (AD) is a new technology that aims to mitigate traffic accidents and enhance driving efficiency. This study presents a deep reinforcement learning (DRL) method for autonomous vehicles that can safely and efficiently handle highway overtaking scenarios. The first step is to create a highway traffic environment where the agent can be guided safely through surrounding vehicles. A hierarchical control framework is then provided to manage high-level driving decisions and low-level control commands, such as speed and acceleration. Next, a special DRL-based method called deep deterministic policy gradient (DDPG) is used to derive decision strategies for use on the highway. The performance of the DDPG algorithm is compared with that of the DQN and PPO algorithms, and the results are evaluated. The simulation results show that the DDPG algorithm can effectively and safely handle highway traffic tasks.

Reinforcement learning (RL), a promising framework for data-driven decision making in an uncertain environment, has successfully been applied in many real-world operation and control problems. However, the application of RL in a large-scale decentralized multi-agent environment remains a challenging problem due to the partial observability and limited communications between agents. In this paper, we develop a model-based kernel RL approach and a model-free deep RL approach for learning a decentralized, shared policy among homogeneous agents. By leveraging the strengths of both these methods, we further propose a novel deep ensemble multi-agent reinforcement learning (MARL) method that efficiently learns to arbitrate between the decisions of the local kernel-based RL model and the wider-reaching deep RL model. We validate the proposed deep ensemble method on a highly challenging real-world air traffic control problem, where the goal is to provide effective guidance to aircraft to avoid air traffic congestion, conflicting situations, and to improve arrival timeliness, by dynamically recommending adjustments of aircraft speeds in real-time. Extensive empirical results from an open-source air traffic management simulation model, developed by Eurocontrol and built on a real-world data set including thousands of aircrafts, demonstrate that our proposed deep ensemble MARL method significantly outperforms three state-of-the-art benchmark approaches.

Deep reinforcement learning for energy trading and load scheduling in residential peer-to-peer energy trading market