A COLREGs-Compliant Collision Avoidance Decision Approach Based on Deep Reinforcement Learning

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

In order to improve the effectiveness of intelligent collision avoidance decision of ships, a collision avoidance decision model is established with the objective of guaranteeing the safety of ship navigation and reducing the loss of voyage, taking real-time data as the input quantity and the requirements of international maritime collision avoidance rules and the common practice of seafarers as the constraints, and MOEA/D (Multi-objective Evolutionary Algorithm Based on Decomposition) algorithm is used to optimize the objective function and calculate the optimal steering angle and steering timing. The simulation results show that the decision can meet the safety and economic requirements of ship collision avoidance under different encounter situations and can provide some reference for ship collision avoidance decision.

In this paper, we propose a novel deep reinforcement learning (DRL) method for optimal path planning for mobile robots using dynamic programming (DP)-based data collection. The proposed method can overcome the slow learning process and improve training data quality inherently in DRL algorithms. The main idea of our approach is as follows. First, we mapped the dynamic programming method to typical optimal path planning problems for mobile robots, and created a new efficient DP-based method to find an exact, analytical, optimal solution for the path planning problem. Then, we used high-quality training data gathered using the DP method for DRL, which greatly improves training data quality and learning efficiency. Next, we established a two-stage reinforcement learning method where, prior to the DRL, we employed extreme learning machines (ELM) to initialize the parameters of actor and critic neural networks to a near-optimal solution in order to significantly improve the learning performance. Finally, we illustrated our method using some typical path planning tasks. The experimental results show that our DRL method can converge much easier and faster than other methods. The resulting action neural network is able to successfully guide robots from any start position in the environment to the goal position while following the optimal path and avoiding collision with obstacles.

COLREGs-abiding hybrid collision avoidance algorithm based on deep reinforcement learning for USVs

During the process of collision avoidance, especially in a multi-ship encounter situation, the dynamic interactions among individual ships impose a significant impact on collision avoidance decision-making. It is imperative, therefore, that collision avoidance decisions are formulated with a comprehensive consideration of not only the current direct collision conflict but also the potential conflicts due to planned collision avoidance actions. To address this requirement, this paper proposes a dynamic conflict cluster detection method for collision avoidance decision-making in multi-ship encounters. The involved ships are clustered into stable temporal-dependent ship conflict groups taking into account both conflict connectivity and the potential spatiotemporal interactions originating from planned collision avoidance actions. The conflict cluster detection model is implemented within a framework to achieve hierarchical coordinated collision avoidance decision-making. By a simulation experiment of an 11-ship encounter, the proposed method successfully discerns the ships with conflicts and provides feasible collision avoidance decisions. Compared to the non-cluster collision avoidance methods, the proposed method generates the results with acceptable deviating distance and number of collision avoidance actions at minimum computation load. It has been demonstrated that the proposed method is both effective and efficient for officers on board and operators at Vessel Traffic Services centers in real-life navigation.

The exploration of ship collision avoidance behavior characteristics can provide a theoretical basis for ship collision risk assessment and collision avoidance decision-making, which is significant for ensuring maritime navigation safety and the development of intelligent ships. In order to scientifically and effectively analyze the characteristics of ship collision-avoidance behavior and to seek the intrinsic connections among ship collision-avoidance behavior feature parameters(CABFPS), this study proposes a method that combines the Apriori algorithm and complex network theory to mine ship collision-avoidance behavior characteristics from massive AIS spatiotemporal data. Based on obtaining ship encounter samples and CABFPS from AIS data, the Apriori algorithm is used to mine the association rules of motion parameters, and the maximum mutual information coefficient is employed to represent the correlation between parameters. Complex networks of CABFPS for different encounter situations are constructed, and network topological indicators are analyzed. Mutual information theory is applied to identify key parameters affecting ship collision- avoidance behavior under different situations. The analysis using actual AIS data indicates that during navigation, the relationships among various parameters are closely linked and prone to mutual influence. The impact of CABFPS on ship collision-avoidance actions varies under different encounter scenarios, with relative distance and DCPA having the greatest influence on ship collision-avoidance actions. This method can comprehensively and accurately mine the correlations between CABFPS and the influence mechanism of parameters on collision-avoidance actions, providing a reference for intelligent ship navigation and the formulation of collision-avoidance decisions.

In the area of artificial intelligence, deep reinforcement learning has grown in significance. It has accomplished extraordinary feats and offers a fresh approach to previously challenging challenges, such as controlling a robotic arm and discovering game strategies. The two primary categories of deep reinforcement learning methods—deep reinforcement learning based on value function and deep reinforcement learning based on policy gradient—are initially explained in this study. The limitations of current approaches and the difficulties faced by deep reinforcement learning methods in related domains are further sorted out, and then the future application directions of deep reinforcement learning methods in the military sphere are examined. Finally, a growing trend for deep reinforcement learning techniques is anticipated in military applications.

The popularity of deep reinforcement learning (DRL) applications in economics has increased exponentially. DRL, through a wide range of capabilities from reinforcement learning (RL) to deep learning (DL), offers vast opportunities for handling sophisticated dynamic economics systems. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this paper, we initially consider a brief review of DL, RL, and deep RL methods in diverse applications in economics, providing an in-depth insight into the state-of-the-art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher efficiency as compared to the traditional algorithms while facing real economic problems in the presence of risk parameters and the ever-increasing uncertainties.

Recently, the development of Low Earth Orbit (LEO) satellites and the advancement of the Mobile Edge Computing (MEC) paradigm have driven the emergence of the Satellite Mobile Edge Computing (Sat-MEC). Sat-MEC has been developed to support communication and task computation for Internet of Things (IoT) Mobile Devices (IMDs) in the absence of terrestrial networks. However, due to the heterogeneity of tasks and Sat-MEC servers, it is still a great challenge to efficiently schedule tasks in Sat-MEC servers. Here, we propose a scheduling algorithm based on the Deep Reinforcement Learning (DRL) method in the Sat-MEC architecture to minimize the average task processing time. We consider multiple factors, including the cooperation between LEO satellites, the concurrency and heterogeneity of tasks, the dynamics of LEO satellites, the heterogeneity of the computational capacity of Sat-MEC servers, and the heterogeneity of the initial queue for task computation. Further, we use the self-attention mechanism to act as a Q-network to extract high-dimensional dynamic information of tasks and Sat-MEC servers. In this work, we model the Sat-MEC environment simulation at the application level and propose a DRL-based task scheduling algorithm. The simulation results confirm the effectiveness of our proposed scheduling algorithm, which reduces the average task processing time by 22.1%, 30.6%, and 41.3%, compared to the genetic algorithm(GA), the greedy algorithm, and the random algorithm, respectively.

The study examines the complex processes of intelligent management of energy resilience in modern healthcare facilities during critical situations, large-scale failures, and prolonged outages of the external centralized power supply. The aim is to develop a comprehensive mathematical model and a Safe Deep Reinforcement Learning (Safe DRL) method that ensures guaranteed compliance with the strict physical and operational constraints of hospital energy systems, even during the intensive training phase of a neural network agent. The objectives are: to formalize in detail the decision-making procedure in an energy system in detail by transitioning to the paradigm of Constrained Markov Decision Processes (CMDP); to develop an innovative mathematical model featuring the implementation of a specialized safety layer based on Lyapunov functions; and to ensure high resilience and autonomy of the system through the implementation of a decentralized Edge-Fog data processing architecture. The methods used include: the theory of Constrained Markov Decision Processes (CMDP), deep reinforcement learning methods based on the Actor-Critic architecture, the mathematical apparatus of Lyapunov stability theory for the analytical correction of actions, and methods of simulation modeling for complex dynamic energy systems. The following results were obtained. In the course of the study, a Safe DRL method was proposed and substantiated, which integrates a Lyapunov-based projection directly into the training loop for the immediate correction of the agent’s control actions. This makes it possible to ensure strict theoretical guarantees of maintaining the required State of Charge (SoC) of battery systems and to prevent critical violations of energy system parameters beyond established safety limits, which is crucial for patient life-support. The effectiveness of the proposed approach was confirmed by a series of numerical experiments in the specialized environment, HospitalEnergyEnv. Under a full blackout scenario, the agent demonstrated adaptability and high accuracy in resource management without any violation of the established physical limits during the entire process of autonomous operation. Conclusions. The scientific novelty of the obtained results lies in the following: the existing optimization model for building energy management systems (BEMS) has been improved by introducing an analytical safety projection mechanism, which minimizes the risks of emergency equipment shutdown during the adaptation of artificial intelligence algorithms; further development of decentralized control methods for critical infrastructure based on Edge-Fog computing has been achieved, which significantly increases system fault tolerance in the event of a loss of connection with the global network and ensures obtaining quasi-optimal solutions in high-dimensional problems. The practical value of this work lies in the potential to create highly reliable autonomous energy systems for critical infrastructure facilities.

The hybrid flow shop scheduling problem (HFSP) has been extensively studied due to its wide applications in modern manufacturing systems. However, existing research primarily focuses on machine-centric scheduling while neglecting the critical role of human resources, whose varying skill levels impact processing efficiency and whose availability constraints complicate scheduling decisions. Therefore, this paper investigates the HFSP with human resource constraints (HFSP-HRC), which requires the joint optimization of job sequencing, machine allocation, and worker assignment to obtain efficient scheduling solutions. We propose a deep reinforcement learning (DRL) method integrated with a meta-path-based heterogeneous graph neural network (MP-HGNN) to solve this problem. It innovatively captures complex interdependencies among operations, machines, and workers through machine-centric and worker-centric meta-paths, overcoming limitations of existing methods that fail to model heterogeneous resource competition relationships. The integration enables high-order semantic feature fusion for enhanced scheduling robustness. Combined with the proximal policy optimization (PPO) algorithm, the framework realizes end-to-end learning from raw scheduling states to optimal decisions. Experiments across four different problem scales demonstrate that the proposed method significantly outperforms baseline heuristic rules and DRL method in makespan, with particularly outstanding performance on large-scale instances.

Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships

With the development of intelligent waterborne transportation, mining collision avoidance patterns based on spatiotemporal and motion data of ships are crucial for the autonomous navigation of intelligent ships, which requires accurate collision avoidance information under various encounter scenarios. Addressing the existing issues of low precision and false detection in data mining algorithms, this paper proposes a collision avoidance behavior mining model considering encounter scenarios. The model is based on the Automatic Identification System (AIS) and the International Regulations for Preventing Collisions at Sea (COLREGs); it firstly identifies ship collision avoidance turning points by analyzing trajectory curvature with turning and recovering factors. Then, by combining AIS data and the specific navigational environment, it matches the ship encounter pairs and determines the encounter scenarios. Comparative experiments show that the model demonstrates superior accuracy in various scenarios compared to traditional algorithm. Finally, the model was applied to AIS data east of the Yangtze River Estuary, recognizing a total of 827 instances of ship collision avoidance behavior under different encounter scenarios. The case study shows that the model can precisely mine collision avoidance information, laying a solid foundation for future research on autonomous collision avoidance decision making for intelligent ships.

The age of algorithms is here, and it is really changing people's lives. More and more ethical problems related to algorithms have attracted people's attention, but the related ethical research is still far behind the research of algorithms. As more intelligent algorithms emerge in an endless stream, there will also be a lot of algorithmic ethical issues. On the other hand, with the continuous improvement of the development level of the automobile industry, people have a stronger demand for the safety and stability of modern transportation, and more and more autonomous driving technology has been promoted and applied in the market. At present, most of the studies on the longitudinal collision avoidance system of vehicles use collision warning or emergency braking to avoid collision. However, when the vehicle is in a special situation such as high speed and slippery road, emergency steering is more effective. In order to further improve the vehicle safety and ethical algorithm design points, this article revolves around vehicle lateral active collision avoidance control method research, the collision avoidance decision-making, and path planning and collision avoidance transverse vehicle longitudinal motion control is analyzed, and based on automated driving simulation experiment, the tests carried out to verify the designed control strategy. The experimental results show that the proposed method not only has a good effect of preventing automatic driving collision but also can meet the requirements of algorithm ethics. This research can effectively guide the research of algorithmic ethics in the field of autonomous driving and effectively reduce the occurrence of traffic accidents.

In this article, a new deep reinforcement learning (RL) method, called asynchronous advantage actor-critic (A3C) method, is developed to solve the optimal control problem of elevator group control systems (EGCSs). The main contribution of this article is that the optimal control law of EGCSs is designed via a new deep RL method, such that the elevator system sends passengers to the desired destination floors as soon as possible. Deep convolutional and recurrent neural networks, which can update themselves during applications, are designed to dispatch elevators. Then, the structure of the A3C method is developed, and the training phase for the learning optimal law is discussed. Finally, simulation results illustrate that the developed method effectively reduces the average waiting time in a complex building environment. Comparisons with traditional algorithms further verify the effectiveness of the developed method.