Crowd-Aware Itinerary Optimization via Clustered Multi-Agent Reinforcement Learning

Recently, the researches on multi-agent reinforcement learning (MARL) have attracted tremendous interest in many applications, especially for autonomous driving. The main problem of MARL is how to deal with the uncertainty in the environment and the interaction between the connected agents. To solve the problem, a distributed robust temporal differential deep Q-network algorithm (MARTD-DQN) was developed in this paper. MARTD-DQN consists of two parts, the decentralized MARL algorithm (DMARL) and the robust TD deep Q-network algorithm (RTD-DQN). DMARL improves the robustness of the policy estimation by fusing the states from the neighbors over communicated networks. RTD- DQN improves the robustness to outliers through on-line estimation of the uncertainty. By combining the two algorithms, the proposed algorithm can be robust not only to node failures but also to the outliers. Then the proposed algorithm is applied to ACC simulations of autonomous cars. The simulation results are given to show the efficiency of the proposed algorithm.

Path planning in an unknown environment is a basic task for mobile robots to complete tasks. As a typical deep reinforcement learning, deep Q-network (DQN) algorithm has gained wide popularity in path planning tasks due to its self-learning and adaptability to complex environment. However, most of path planning algorithms based on DQN spend plenty of time for model training and the learned model policy depends only on the information observed by sensors. It will cause poor generalization capability for the new task and time waste for model retraining. Therefore, a new deep reinforcement learning method combining DQN with prior knowledge is proposed to reduce training time and enhance generalization capability. In this method, a fuzzy logic controller is designed to avoid the obstacles and help the robot avoid blind exploration for reducing the training time. A target-driven approach is used to address the lack of generalization, in which the learned policy depends on the fusion of observed information and target information. Extensive experiments show that the proposed algorithm converges faster than DQN algorithm in path planning tasks and the target can be reached without retraining when the path planning task changes.

Smart transportation networks are challenged by a critical challenge of traffic congestion and inevitably requires a complex package of management solutions. This paper introduces a Secure Multi-Agent Reinforcement Learning (MARL) framework that optimizes traffic whilst protecting from malicious threats. Traffic Prediction Dataset accessible through Kaggle is used as the foundation for training reinforcement learning agents from four junctions who cooperate to manage traffic signals. The widespread preprocessing procedures that also dealt with data gaps in addition to dimension transformation and value normalization generated genuine traffic prediction models. The Multi-Agent Reinforcement Learning framework reduced the mean waiting time by 45.7% with a 47.0% traffic passing rate higher than that achieved using standard fixed-duration signal control techniques. Homomorphic encryption, when integrated with blockchain protocols, instituted novel security protocols for agent to agent communication, and this yielded a reduction of adversarial attacks below 90% levels. The implementation framework provided a 39.3% reduction in the fuel consumption when implemented in real-time urban traffic management systems. Secure Multi-Agent Reinforcement Learning (MARL) systems show their potential aptitude to propel smart transportation networks in the while providing increased efficiency and security as well as sustainability effectiveness.

In this paper, we investigate the user pairing and power allocation scheme for multiple cellular users (CUs) under the downlink non-orthogonal multiple access (NOMA) system. To maximize the sum-rate of all CUs, we formulate the resource allocation issue through optimizing the user pairing relationship and power allocation method. However, due to the nonconvex property of the formulated problem, the original problem is decoupled into two subproblems. First, the optimal power allocation scheme with a given subchannel assignment is obtained via a closed-form solution. Furthermore, based on the obtained optimal allocation scheme, a classical deep reinforcement learning (DRL) method called Deep Q-Network (DQN) algorithm is adopted to find the optimal user pairing scheme, where the DQN algorithm is characterized by higher learning efficiency and better performance of the features extraction ability compared with traditional reinforcement learning (RL) schemes. Simulation results validate the effectiveness of our proposed resource allocation, as compared against the RL based scheme and conventional orthogonal multiple access (OMA) method.

Financial institutions are increasingly dealing with difficulties in preserving transaction integrity due to escalating digital fraud and extensive cloud-based operations. Conventional machine learning and rule-based detection systems frequently have latency in replies, restricted adaptability, and an absence of verifiable audit trails. The proposed study presents a Hybrid Multi-Agent Reinforcement Learning (MARL) and Blockchain framework to overcome existing limitations, integrating adaptive multi-agent coordination for real-time fraud detection with a permissioned blockchain ledger to guarantee secure, tamper-proof validation of agent decisions. MARL agents adaptively acquire optimal detection strategies through incentive shaping and inter-agent communication, while blockchain consensus and smart contracts substantiate decision proofs for regulatory adherence. The experimental evaluation of the Kaggle Financial Transactions dataset demonstrates improved accuracy relative to baseline models: Accuracy 0.989, Precision 0.981, Recall 0.987, F1-Score 0.984, AUC-ROC 0.992, with negligible latency (1.8 ms/transaction). The comparative research reveals a persistent superiority over Random Forest, XGBoost, CNN- LSTM, and Transformer-based classifiers, showcasing robustness, scalability, and real-time application. The suggested study enhances the area by merging reinforcement learning adaptability with blockchain security, establishing a realistic, high-throughput, and reliable framework for cloud- based financial systems, while presenting a model for future research in agentic AI-driven fraud prevention.

A Multi-Agent Reinforcement Learning Framework for Public Health Decision Analysis.

Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment. Reinforcement Learning methods enable a state action function in mobile robots suited to their environment. During trial-and-error interaction with its surroundings, it helps a robot to find an ideal behavior on its own. The Deep Q Network (DQN) algorithm is used in TurtleBot 3 (TB3) to achieve the goal by successfully avoiding the obstacles. But it requires a large number of training iterations. This research mainly focuses on a mobility robot’s best path prediction utilizing DQN and the Artificial Potential Field (APF) algorithms. First, a TB3 Waffle Pi DQN is built and trained to reach the goal. Then the APF shortest path algorithm is incorporated into the DQN algorithm. The proposed planning approach is compared with the standard DQN method in a virtual environment based on the Robot Operation System (ROS). The results from the simulation show that the combination is effective for DQN and APF gives a better optimal path and takes less time when compared to the conventional DQN algorithm. The performance improvement rate of the proposed DQN + APF in comparison with DQN in terms of the number of successful targets is attained by 88%. The performance of the proposed DQN + APF in comparison with DQN in terms of average time is achieved by 0.331 s. The performance of the proposed DQN + APF in comparison with DQN average rewards in which the positive goal is attained by 85% and the negative goal is attained by −90%.

Path planning is a key task in mobile robots, and the application of Deep Q Network (DQN) algorithm for mobile robot path planning has become a hotspot and challenge in current research. In order to solve the obstacle avoidance limitations faced by the DQN algorithm in indoor robot path planning, this paper proposes a solution based on an improved DQN algorithm. In view of the low learning efficiency of the DQN algorithm, the Duel DQN structure is introduced to enhance the performance and combined with a Prioritized Experience Replay (PER) mechanism to ensure the stability of the robot during the learning process. In addition, the idea of Munchausen Deep Q Network (M-DQN) is incorporated to guide the robot to learn the optimal policy more effectively. Based on the above improvements, the PER-D2MQN algorithm is proposed in this paper. In order to validate the effectiveness of the proposed algorithm, we conducted multidimensional simulation comparison experiments of the PER-D2MQN algorithm with DQN, Duel DQN, and the existing methodology PMR-DQN in the Gazebo simulation environment and examined the cumulative and average rewards for reaching the goal point, the number of convergent execution steps, and the time consumed by the robot in reaching the goal point. The simulation results show that the PER-D2MQN algorithm obtains the highest reward in both static and complex environments, exhibits the best convergence, and finds the goal point with the lowest average number of steps and the shortest elapsed time.

This study proposes a deep Q-network (DQN) model for electric motorcycles (EMs) and a multi-agent reinforcement learning (MARL)-based central control system to support battery swapping decision-making in the delivery business. We aim to minimize expected delivery losses, especially in scenarios where delivery requests are randomly and independently generated for each EM, with fluctuating time distributions and limited BSS capacity. Our MARL benefits from a reservation mechanism and a profit-aggregated central system, which greatly reduces the complexity of MARL. Furthermore, to address the inherent non-stationary problems of MARL, we propose a decentralized agent-based MARL framework, named Decentralized Agents, Centralized Learning Deep Q Network. This framework, leveraging a tailored learning algorithm, achieves peak-averse behavior, reducing delivery losses. Additionally, we introduce a hybrid approach that combines the resulting DQN algorithm for determining when to visit the BSS, and a greedy algorithm for deciding which BSS to visit. Computational experiments using real-world delivery data are conducted to evaluate the performance of our algorithm. The results demonstrate that the hybrid approach maximizes the overall profit of the entire EM fleet in a challenging environment with limited BSS capacity.

The Micro-Electro-Mechanical Systems Fabry-Perot (MEMS-FP) filter is a pivotal photosensitive component within optical instrumentation, serving an integral role across various optical applications, including frequency selection spectroscopy and spectral analysis. The advent of MEMS-FP technology has significantly contributed to advancements in the optical field, underscoring the necessity for precise pre-fabrication theoretical predictions of device properties through finite element analysis (FEA). Nonetheless, the complexity inherent in three-dimensional modelling of MEMS-FP structures, coupled with the intricate nature of finite element simulations, presents substantial challenges that impede progress in MEMS-FP research. To navigate these challenges, this paper introduces a novel approach employing reinforcement learning (RL) for the automated design of MEMS-FP devices. This method encompasses the development of a Deep Q-Network (DQN) algorithm, the formulation of a tailored reward function, and an innovative strategy for updating the status of MEMS-FP devices. Various designs schemes of MEMS-FP devices can be acquired using MEMS-FP state search. The disparity between the MEMS-FP device design scheme and performance index can be assessed using a reward function. The DQN algorithm is crafted to update the network responsible for predicting Q-values. The agent trained by our method can rapidly calculate the MEMS-FP device design scheme that fulfils a specified performance index when provided with the performance index of the MEMS-FP device. By amalgamating reinforcement learning with MEMS-FP device research, this approach adeptly identifies optimal design configurations that meet specified performance criteria with enhanced precision and efficiency and promote the advancement of MEMS-FP devices.

The semiconductor industry is a resource-intensive sector that heavily relies on energy, water, chemicals, and raw materials. Within the semiconductor manufacturing process, the diffusion furnace, ion implantation machine, and plasma etching machine exhibit high energy demands or operate at extremely high temperatures, resulting in significant electricity consumption, which is usually carbon-intensive. To address energy conservation concerns, the industry adopts batch production technology, which allows for the simultaneous processing of multiple products. The energy-efficient parallel batch scheduling problem arises from the need to optimise product grouping and sequencing. In contrast to existing heuristics, meta-heuristics, and exact algorithms, this paper introduces the Deep Q-Network (DQN) algorithm as a novel approach to address the proposed problem. The DQN algorithm is built upon the agent’s systematic learning of scheduling rules, thereby enabling it to offer guidance for online decision-making regarding the grouping and sequencing of products. The efficacy of the algorithm is substantiated through extensive computational experiments.

3D ultrasound (US) has become prevalent due to its rich spatial and diagnostic information not contained in 2D US. Moreover, 3D US can contain multiple standard planes (SPs) in one shot. Thus, automatically localizing SPs in 3D US has the potential to improve user-independence and scanning-efficiency. However, manual SP localization in 3D US is challenging because of the low image quality, huge search space and large anatomical variability. In this work, we propose a novel multi-agent reinforcement learning (MARL) framework to simultaneously localize multiple SPs in 3D US. Our contribution is four-fold. First, our proposed method is general and it can accurately localize multiple SPs in different challenging US datasets. Second, we equip the MARL system with a recurrent neural network (RNN) based collaborative module, which can strengthen the communication among agents and learn the spatial relationship among planes effectively. Third, we explore to adopt the neural architecture search (NAS) to automatically design the network architecture of both the agents and the collaborative module. Last, we believe we are the first to realize automatic SP localization in pelvic US volumes, and note that our approach can handle both normal and abnormal uterus cases. Extensively validated on two challenging datasets of the uterus and fetal brain, our proposed method achieves the average localization accuracy of 7.03 degrees/1.59mm and 9.75 degrees/1.19mm. Experimental results show that our light-weight MARL model has higher accuracy than state-of-the-art methods.

Efficient parking distribution is crucial for urban traffic management; nevertheless, variable demand and spatial disparities raise considerable obstacles. Current research emphasizes local optimization but neglects the fundamental challenges of real-time parking allocation, resulting in inefficiencies within intricate metropolitan settings. This research delineates two key issues: (1) A dynamic imbalance between supply and demand, characterized by considerable fluctuations in parking demand over time and across different locations, rendering static allocation solutions inefficient; (2) spatial resource optimization, aimed at maximizing the efficiency of limited parking spots to improve overall system performance and user satisfaction. We present a Multi-Agent Reinforcement Learning (MARL) framework that incorporates adaptive optimization and intelligent collaboration for dynamic parking allocation to tackle these difficulties. A reinforcement learning-driven temporal decision mechanism modifies parking assignments according to real-time data, whilst a Graph Neural Network (GNN)-based spatial model elucidates inter-parking relationships to enhance allocation efficiency. Experiments utilizing actual parking data from Melbourne illustrate that Multi-Agent Reinforcement Learning (MARL) substantially surpasses conventional methods (FIFO, SIRO) in managing demand variability and optimizing resource distribution. A thorough quantitative investigation confirms the strength and flexibility of the suggested method in various urban contexts.

Hybrid beamforming with massive multiple-input multiple-output (MIMO) antenna arrays plays a pivotal role in achieving high data rates and spectral efficiency in next-generation wireless systems. Nevertheless, the real-time 62design and adaptation of beamforming codebooks and user grouping remain challenging, as conventional machine learning (ML) methods typically rely on the centralized collection of extensive channel state information (CSI), resulting in substantial feedback overhead and potential privacy violations. This chapter proposes a federated multi-agent reinforcement learning (MARL) framework for joint hybrid beamforming and dynamic user grouping. Unlike prior work which focuses solely on centralized codebook learning, the proposed approach enables distributed user equipment (UE) to collaboratively train codebook and clustering algorithms without sharing raw data, thereby preserving privacy. The proposed method significantly reduces communication overhead while achieving near-optimal beam patterns and robust scalability in dense network scenarios, making it well-suited for practical deployment in next-generation wireless systems. The experimental findings indicate that for mobile users, federated learning (FL)-based user clustering with MARL outperforms traditional initial access-based beam clustering in MARL.

Time-triggered Ethernet combines time-triggered and event-triggered communication, and is suitable for fields with high real-time requirements. Aiming at the problem that the traditional scheduling algorithm is not effective in scheduling event-triggered messages, a message scheduling algorithm based on multi-agent reinforcement learning (MADDPG, Multi-Agent Deep Deterministic Policy Gradient) and a hybrid algorithm combining SMT (Satisfiability Modulo Theories) solver and MADDPG are proposed. This method aims to optimize the scheduling of event-triggered messages while maintaining the uniformity of time-triggered message scheduling, providing more time slots for event-triggered messages, and reducing their waiting time and end-to-end delay. Through the designed scheduling software, in the experiment, compared with the SMT-based algorithm and the traditional DQN (Deep Q-Network) algorithm, the new method shows better load balance and lower message jitter, and it is verified in the OPNET simulation environment that it can effectively reduce the delay of event-triggered messages.