Characterizing Attacks on Deep Reinforcement Learning

The intersection management system can increase traffic capacity, vehicle safety, and the smoothness of all vehicle movement. Platoons of connected vehicles (CVs) use communication technologies to share information with each other and with infrastructures. In this paper, we proposed a deep reinforcement learning (DRL) model that applies to vehicle platooning at an isolated signalized intersection with partial detection. Moreover, we identified hyperparameters and tested the system with different numbers of vehicles (1, 2, and 3) in the platoon. To compare the effectiveness of the proposed model, we implemented two benchmark options, actuated traffic signal control (ATSC) and max pressure (MP). The experimental results demonstrated that the DRL model has many outstanding advantages compared to other models. Through the learning process, the average waiting time of vehicles in the DRL method was improved by 20% and 28% compared with the ATSC and MP options. The results also suggested that the DRL model is effective when the CV penetration rate is over 20%.

This paper investigates stock portfolios by application of Deep Reinforcement Learning (DRL) Models to achieve an optimal tactical asset allocation. The research problem is described as an optimization scenario that seeks to maximize the portfolio risk adjusted returns for a given portfolio asset allocation. The problem is set up with an initial capital investment which is invested in a set of assets. The initial strategic allocation is determined, which in our case is the equal weight allocation, and all of the capital is invested in the set of assets. At each point in time, the assets are reallocated according to the allocation which will increase the portfolio value. Two DRL models are implemented. The performance of the DRL models is compared with the uniform weights portfolio. The results show that, generally, two DRL models have higher cumulative returns.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

Irrigation decision-making using Reinforcement Learning (RL) performs well in changing environment, but easily falls into sub-optimal solutions with high-dimensional data. Deep Reinforcement Learning (DRL) has fused RL with Deep Learning (DL) and excels at learning adaptive and long-term irrigation strategies directly from high-dimensional environment data. This paper systematically reviews the applications of DRL in irrigation optimization, covering both pre-trained environments based on crop growth simulators and dynamic environments driven by real-time sensors. We discussed the strengths of classic DRL algorithms, including their ability to handle dynamic and non-linear environments, and reviewed their performance in irrigation multi-objective optimization and decision-making. In addition, we identified constraints in applying DRL in irrigation decision making, which include data scarcity, poor model interpretability, and difficulties in field deployment. It shows DRL can provide a powerful framework for adaptive irrigation, but is constrained by the gap between simulation and real-world complexity. To address these limitations, we discussed approaches in future work, such as developing multi-objective DRL algorithms. These approaches will improve DRL modeling outcomes and provide a technological foundation for smart agriculture and sustainable resource management. • Review the application of Deep Reinforcement Learning (DRL) in agricultural irrigation. • Analyze the performance of DRL algorithms in irrigation decision-making. • Compare DRL models based on different environment in irrigation optimization. • Discuss the further work to improve DRL performance in irrigation optimization.

Intrusion detection has been a vast-surveyed topic for many decades as network attacks are tremendously growing. This has heightened the need for security in networks as web-based communication systems are advanced nowadays. The proposed work introduces an intelligent semi-supervised intrusion detection system based on different algorithms to classify the network attacks accurately. Initially, the pre-processing is accomplished using null value dropping and standard scaler normalization. After pre-processing, an enhanced Deep Reinforcement Learning (EDRL) model is employed to extract high-level representations and learn complex patterns from data by means of interaction with the environment. The enhancement of deep reinforcement learning is made by associating a deep autoencoder (AE) and an improved flamingo search algorithm (IFSA) to approximate the Q-function and optimal policy selection. After feature representations, a support vector machine (SVM) classifier, which discriminates the input into normal and attack instances, is employed for classification. The presented model is simulated in the Python platform and evaluated using the UNSW-NB15, CICIDS2017, and NSL-KDD datasets. The overall classification accuracy is 99.6%, 99.93%, and 99.42% using UNSW-NB15, CICIDS2017, and NSL-KDD datasets, which is higher than the existing detection frameworks.

Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

With the spread of online platforms, problems such as manned/unmanned food delivery, cargo delivery, raw material delivery, are increasing the importance of logistics day by day. Vehicle routing problem, which is one of the most important problems in the field of logistics, is a combinatorial problem and as the problem space grows, it takes a long time to find a solution with human effort and in most cases it is not even possible. Thus, it becomes essential for the solution of this problem to be autonomous. Although it is possible to solve the problem with classical heuristic optimization methods, it takes a long time and sometimes does not give a good enough solution. Deep reinforcement learning models with attention mechanisms have great potential in this regard. However,in case of insufficient training in large problem space, it is possible to get away from the optimal solution. In this study, better results are taken in an acceptable time by using the deep reinforcement learning models with attention-model and heuristic methods in a hybrid way.

Vision-based object positioning is very important in the electronic industry for assembly and inspection tasks. Many methods have been proposed to tackle the problem, either by traditional machine vision or by deep learning (DL) techniques. The traditional methods rely on template matching or feature point correspondence. They are computationally intensive and are easily affected by illumination changes and noise. DL models such as convolutional neural networks (CNNs) are computationally very efficient but are also sensitive against environmental changes. In this article, a deep reinforcement learning (DRL) model based on the Actor-Critic style Proximal Policy Optimization algorithm(s) (AC-PPO) is proposed. The proposed method is applied for the positioning of printed circuit boards (PCBs). The model uses as the current environment the sensed image and the reference template as a guide. It requires only a single manually marked template in the reference image. All possible training images are automatically and randomly generated during the neural network training without human intervention. The proposed reinforcement learning (RL) model is shown to be adaptive to environmental changes, including illumination, noise, de-focusing, and template occlusion, compared with the CNN regressor. Experimental results indicate that the proposed model on average can achieve estimation errors less than 1 pixel in translation and 1° in orientation, with fast evaluation for the real-time PCB positioning task.

The increasing usage of Unmanned Aerial Vehicles (UAVs) in diverse applications necessitates the development of effective flight control systems. A major difficulty, nevertheless, is maintaining robust flight control under complex and dynamic environmental conditions, such as obstacle and non-flying zones, wind disturbances, and sensor noise. Traditional control techniques fail to achieve required flight control in these environments. To address this point, an adaptive control strategy is proposed based on a Deep Reinforcement Learning (DRL) model to enhance the flight performance of quadcopters. The learning and adaptation of the DRL-based control strategy are implemented in real-time through continuous interaction with the environment. This is to improve flight control and achieve consistent UAV performance under varying conditions. Proximal policy optimization with a reward function is used to minimize positional errors, ensure collision-free flight paths, and reduce energy consumption. The developed DRL model for quadcopters is trained in a simulated environment and then tested in three complex environmental scenarios, including urban, forest and mountain terrains. Experimental results demonstrate remarkable improvements in UAV flight performance. In the training phase, the reported training reward increased from 10 to 110 and the train loss is dropped from 0.85 to 0.05, which indicated successful model learning. Also, during system verification, rewards increased from 12 to 115 and UAV flight path deviations were decreased from 0.5 to 0.08m. The proposed controller outperforms conventional approaches in urban environments by lowering average trajectory deviations to 0.2m from 0.35m for MPC and 0.6m for PID. Also, the developed DRL-based controller outperformed the PID and MPC controllers, with path deviations of 0.18m in mountains and 0.12m in forests. In addition, fewer collision rates with obstacles are achieved with the model, 3% in forest, 1.8% in urban areas, and 4.5% in mountains. Furthermore, the consumed energy is reduced to 950J as compared to 1200J for PID and 1050J for MPC. The results show the strength of deploying the proposed controller in following the intended UAV flight path with high precision, effectively avoiding detected conflicts and minimizing consumed energy by the UAV.

Bridging the simulation-to-real gap of depth images for deep reinforcement learning

Study on deep reinforcement learning techniques for building energy consumption forecasting

Various factors make electricity markets increasingly complex, making their analysis challenging. This complexity demands advanced analytical tools to manage and understand market dynamics. This paper explores the application of deep reinforcement learning (DRL) and bi-level optimization models to analyze and simulate electricity markets. We introduce a bi-level optimization framework incorporating realistic market constraints, such as non-convex operational characteristics and binary decision variables, to establish an upper-bound benchmark for evaluating the performance of DRL algorithms. The results confirm that DRL methods do not reach the theoretical upper bounds set by the bi-level models, thereby confirming the effectiveness of the proposed model in providing a clear performance target for DRL. This benchmarking approach demonstrates DRL's current capabilities and limitations in complex market environments but also aids in developing more effective DRL strategies by providing clear, quantifiable targets for improvement. The proposed method can also identify the information gap cost since DRL methods operate under more realistic conditions than optimization techniques, given that they don't need to assume complete knowledge about the system. This study thus provides a foundation for future research to enhance market understanding and possibly its efficiency in the face of increasing complexity in the electricity market. Our methodology's effectiveness is further validated through a large-scale case study involving 150 power plants, demonstrating its scalability and applicability to real-world scenarios.

In this article, the authors attempt to extend human feedback as a critical decision-making tool to improve DRL for RPA systems, especially in application domains sensitive to change, such as healthcare and legal practice. The previous methods of implementing RPA moved in well-defined trajectories, meaning they have low adaptability for complex decision-making or other atypical cases. These systems can integrate DRL and hence can learn and evolve over the period required. Nevertheless, DRL models often work independently from human beings. They are used in applications with low human interaction, which can cause issues in decision-making procedures that require a higher understanding of regulation. The article puts human-in-the-loop, with human experts returning to the DRL models. It encapsulates human feedback into meaningful rewards or penalties for the DRL algorithms, ensuring correction and improvement of decision-making brought about by the expert’s input. This approach can enhance the management of exceptions/corner cases typical to regulated domains, the automation of which must consider compliance and legal requirements. The paper focuses on how human intervention can be combined in DRL to augment the reward schemes that are superior to those provided by purely automated systems to human overseers. The efficiency of the hybrid system of DRL coupled with RPA in real-world use cases that involve processing legal documents and medical records is well supported. Currently, feedback from people in DRL for RPA has the potential to create more dynamic, dependable, and adherent robotic processes in various stringent compliance contexts.

Deep reinforcement learning (DRL) can be used to extract deep features that can be incorporated into reinforcement learning systems to enable improved decision-making; DRL can therefore also be used for managing stock portfolios. Traditional methods cannot fully exploit the advantages of DRL because they are generally based on real-time stock quotes, which do not have sufficient features for making comprehensive decisions. In this study, in addition to stock quotes, we introduced stock financial indices as additional stock features. Moreover, we used Markowitz mean-variance theory for determining stock correlation. A three-agent deep reinforcement learning model called Collaborative Multi-agent reinforcement learning-based stock Portfolio management System (CMPS) was designed and trained based on fused data. In CMPS, each agent was implemented with a deep Q-network to obtain the features of time-series stock data, and a self-attention network was used to combine the output of each agent. We added a risk-free asset strategy to CMPS to prevent risks and referred to this model as CMPS-Risk Free (CMPS-RF). We conducted experiments under different market conditions using the stock data of China Shanghai Stock Exchange 50 and compared our model with the state-of-the-art models. The results showed that CMPS could obtain better profits than the compared benchmark models, and CMPS-RF was able to accurately recognize the market risk and achieved the best Sharpe and Calmar ratios. The study findings are expected to aid in the development of an efficient investment-trading strategy.