Efficient congestion control in communications using novel weighted ensemble deep reinforcement learning

Congestion control in Cluster Wireless Sensor Networks (CWSNs) has drawn widespread attention and research interests. The increasing number of nodes and scale of networks cause more complex congestion control and management. Active Queue Management (AQM) is one of the major congestion control approaches in CWSNs, and Random Early Detection (RED) algorithm is commonly used to achieve high utilization in AQM. However, traditional RED algorithm depends exclusively on source-side control, which is insufficient to maintain efficiency and state stability. Specifically, when congestion occurs, deficiency of feedback will hinder the instability of the system. In this paper, we adopt the Additive-Increase Multiplicative-Decrease (AIMD) adjustment scheme and propose an improved RED algorithm by using neighbor feedback and scheduling scheme. The congestion control model is presented, which is a linear system with a non-linear feedback, and modeled by Lur’e type system. In the context of delayed Lur’e dynamical network, we adopt the concept of cluster synchronization and show that the congestion controlled system is able to achieve cluster synchronization. Sufficient conditions are derived by applying Lyapunov-Krasovskii functionals. Numerical examples are investigated to validate the effectiveness of the congestion control algorithm and the stability of the network.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

The intersection management system can increase traffic capacity, vehicle safety, and the smoothness of all vehicle movement. Platoons of connected vehicles (CVs) use communication technologies to share information with each other and with infrastructures. In this paper, we proposed a deep reinforcement learning (DRL) model that applies to vehicle platooning at an isolated signalized intersection with partial detection. Moreover, we identified hyperparameters and tested the system with different numbers of vehicles (1, 2, and 3) in the platoon. To compare the effectiveness of the proposed model, we implemented two benchmark options, actuated traffic signal control (ATSC) and max pressure (MP). The experimental results demonstrated that the DRL model has many outstanding advantages compared to other models. Through the learning process, the average waiting time of vehicles in the DRL method was improved by 20% and 28% compared with the ATSC and MP options. The results also suggested that the DRL model is effective when the CV penetration rate is over 20%.

Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to adversarial attacks, which attack DRL models by adding small perturbations to the observations. However, some attacks assume full availability of the victim model, and some require a huge amount of computation, making them less feasible for real world applications. In this work, we make further explorations of the vulnerabilities of DRL by studying other aspects of attacks on DRL using realistic and efficient attacks. First, we adapt and propose efficient black-box attacks when we do not have access to DRL model parameters. Second, to address the high computational demands of existing attacks, we introduce efficient online sequential attacks that exploit temporal consistency across consecutive steps. Third, we explore the possibility of an attacker perturbing other aspects in the DRL setting, such as the environment dynamics. Finally, to account for imperfections in how an attacker would inject perturbations in the physical world, we devise a method for generating a robust physical perturbations to be printed. The attack is evaluated on a real-world robot under various conditions. We conduct extensive experiments both in simulation such as Atari games, robotics and autonomous driving, and on real-world robotics, to compare the effectiveness of the proposed attacks with baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.

This paper investigates stock portfolios by application of Deep Reinforcement Learning (DRL) Models to achieve an optimal tactical asset allocation. The research problem is described as an optimization scenario that seeks to maximize the portfolio risk adjusted returns for a given portfolio asset allocation. The problem is set up with an initial capital investment which is invested in a set of assets. The initial strategic allocation is determined, which in our case is the equal weight allocation, and all of the capital is invested in the set of assets. At each point in time, the assets are reallocated according to the allocation which will increase the portfolio value. Two DRL models are implemented. The performance of the DRL models is compared with the uniform weights portfolio. The results show that, generally, two DRL models have higher cumulative returns.

Study on deep reinforcement learning techniques for building energy consumption forecasting

Irrigation decision-making using Reinforcement Learning (RL) performs well in changing environment, but easily falls into sub-optimal solutions with high-dimensional data. Deep Reinforcement Learning (DRL) has fused RL with Deep Learning (DL) and excels at learning adaptive and long-term irrigation strategies directly from high-dimensional environment data. This paper systematically reviews the applications of DRL in irrigation optimization, covering both pre-trained environments based on crop growth simulators and dynamic environments driven by real-time sensors. We discussed the strengths of classic DRL algorithms, including their ability to handle dynamic and non-linear environments, and reviewed their performance in irrigation multi-objective optimization and decision-making. In addition, we identified constraints in applying DRL in irrigation decision making, which include data scarcity, poor model interpretability, and difficulties in field deployment. It shows DRL can provide a powerful framework for adaptive irrigation, but is constrained by the gap between simulation and real-world complexity. To address these limitations, we discussed approaches in future work, such as developing multi-objective DRL algorithms. These approaches will improve DRL modeling outcomes and provide a technological foundation for smart agriculture and sustainable resource management. • Review the application of Deep Reinforcement Learning (DRL) in agricultural irrigation. • Analyze the performance of DRL algorithms in irrigation decision-making. • Compare DRL models based on different environment in irrigation optimization. • Discuss the further work to improve DRL performance in irrigation optimization.

Intrusion detection has been a vast-surveyed topic for many decades as network attacks are tremendously growing. This has heightened the need for security in networks as web-based communication systems are advanced nowadays. The proposed work introduces an intelligent semi-supervised intrusion detection system based on different algorithms to classify the network attacks accurately. Initially, the pre-processing is accomplished using null value dropping and standard scaler normalization. After pre-processing, an enhanced Deep Reinforcement Learning (EDRL) model is employed to extract high-level representations and learn complex patterns from data by means of interaction with the environment. The enhancement of deep reinforcement learning is made by associating a deep autoencoder (AE) and an improved flamingo search algorithm (IFSA) to approximate the Q-function and optimal policy selection. After feature representations, a support vector machine (SVM) classifier, which discriminates the input into normal and attack instances, is employed for classification. The presented model is simulated in the Python platform and evaluated using the UNSW-NB15, CICIDS2017, and NSL-KDD datasets. The overall classification accuracy is 99.6%, 99.93%, and 99.42% using UNSW-NB15, CICIDS2017, and NSL-KDD datasets, which is higher than the existing detection frameworks.

In recent years, the recommendation system and robot learning are undoubtedly the two most popular application fields, and the core algorithms supporting these two fields are deep learning based on perception and reinforcement learning based on exploration learning, respectively. How to combine these two fields to better improve the development of the whole machine learning field is the dream of numerous researchers. The Deep Reinforcement Network (DRN) model successfully embedded reinforcement learning into the recommendation system, which provided a good idea for subsequent researchers. However, the disadvantage is also obvious, that is, the DRN model is built for news recommendations, meaning that the DRN model is not transferable, which is also the defect of many current recommendation system models. Meanwhile, the agent learning method adopted by the DRN model is primitive and inefficient. Among many models and algorithms that have emerged in recent years, we use the newly proposed deployment efficiency to measure their comprehensive quality and found that few models focus on both efficiency and performance improvement. To fill the gap of model deployment efficiency neglected by many researchers and to create a model of reinforcement learning agents with stronger performance, we have been exploring and trying to complete research on the Gate Attentional Factorization Machines (GAFM) model. Finally, we successfully integrated the GAFM model and reinforcement learning. The Deep Reinforcement Factorization Machines (DRFM) model proposed in this paper is based on the combination of deep learning with high perception ability and reinforcement learning with high exploration ability, centered on improving the deployment efficiency and learning performance of the model. The GAFM model is modified and upgraded using multidisciplinary techniques, and a new model-based random exploration strategy is proposed to update and optimize the recommendation list efficiently. Through parallel contrast experiments on various datasets, it is proved that the DRFM model surpasses the traditional recommendation system model in all aspects. The DRFM model is far superior to other models in terms of performance and robustness, and also significantly improved in terms of deployment efficiency. At the same time, we conduct a comparative analysis with the latest deep reinforcement learning algorithm and prove the unique advantages of the DRFM model.

Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading

A new ensemble deep graph reinforcement learning network for spatio-temporal traffic volume forecasting in a freeway network

Job shop scheduling problem (JSSP) is one of the well‐known NP‐hard combinatorial optimization problems (COPs) that aims to optimize the sequential assignment of finite machines to a set of jobs while adhering to specified problem constraints. Conventional solution approaches which include heuristic dispatching rules and evolutionary algorithms has been largely in use to solve JSSPs. Recently, the use of reinforcement learning (RL) has gained popularity for delivering better solution quality for JSSPs. In this research, we propose an end‐to‐end deep reinforcement learning (DRL) based scheduling model for solving the standard JSSP. Our DRL model uses attention‐based encoder of Transformer network to embed the JSSP environment represented as a disjunctive graph. We introduced Gate mechanism to modulate the flow of learnt features by preventing noise features from propagating across the network to enrich the representations of nodes of the disjunctive graph. In addition, we designed a novel Gate‐based graph pooling mechanism that preferentially constructs the graph embedding. A simple multi‐layer perceptron (MLP) based action selection network is used for sequentially generating optimal schedules. The model is trained using proximal policy optimization (PPO) algorithm which is built on actor critic (AC) framework. Experimental results show that our model outperforms existing heuristics and state of the art DRL based baselines on generated instances and well‐known public test benchmarks. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Reinforcement learning (RL), a promising framework for data-driven decision making in an uncertain environment, has successfully been applied in many real-world operation and control problems. However, the application of RL in a large-scale decentralized multi-agent environment remains a challenging problem due to the partial observability and limited communications between agents. In this paper, we develop a model-based kernel RL approach and a model-free deep RL approach for learning a decentralized, shared policy among homogeneous agents. By leveraging the strengths of both these methods, we further propose a novel deep ensemble multi-agent reinforcement learning (MARL) method that efficiently learns to arbitrate between the decisions of the local kernel-based RL model and the wider-reaching deep RL model. We validate the proposed deep ensemble method on a highly challenging real-world air traffic control problem, where the goal is to provide effective guidance to aircraft to avoid air traffic congestion, conflicting situations, and to improve arrival timeliness, by dynamically recommending adjustments of aircraft speeds in real-time. Extensive empirical results from an open-source air traffic management simulation model, developed by Eurocontrol and built on a real-world data set including thousands of aircrafts, demonstrate that our proposed deep ensemble MARL method significantly outperforms three state-of-the-art benchmark approaches.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

Congestion is an important issue which researchers focus on in the Transmission Control Protocol (TCP) network environment. To keep the stability of the whole network, congestion control algorithms have been extensively studied. Queue management method employed by the routers is one of the important issues in the congestion control study. Active Queue Management (AQM) has been proposed as a router-based mechanism for early detection of congestion inside the network. In this study, we are comparing AQM two popular queue management methods, Random Early Detection (RED) and droptail, in different aspects, such as throughput and fairness Index. The comparison results indicate RED performed slightly better with higher throughput and higher fairness Index than droptail. Simulation is done by using Network Simulator (NS2) and the graphs are drawn using X- graph.