Research on Intelligent Signal Timing Optimization of Signalized Intersection Based on Deep Reinforcement Learning Using Floating Car Data

Research on the Real-time Traffic Status Identification of Signalized Intersections Based on Floating Car Data

The intersection management system can increase traffic capacity, vehicle safety, and the smoothness of all vehicle movement. Platoons of connected vehicles (CVs) use communication technologies to share information with each other and with infrastructures. In this paper, we proposed a deep reinforcement learning (DRL) model that applies to vehicle platooning at an isolated signalized intersection with partial detection. Moreover, we identified hyperparameters and tested the system with different numbers of vehicles (1, 2, and 3) in the platoon. To compare the effectiveness of the proposed model, we implemented two benchmark options, actuated traffic signal control (ATSC) and max pressure (MP). The experimental results demonstrated that the DRL model has many outstanding advantages compared to other models. Through the learning process, the average waiting time of vehicles in the DRL method was improved by 20% and 28% compared with the ATSC and MP options. The results also suggested that the DRL model is effective when the CV penetration rate is over 20%.

Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to adversarial attacks, which attack DRL models by adding small perturbations to the observations. However, some attacks assume full availability of the victim model, and some require a huge amount of computation, making them less feasible for real world applications. In this work, we make further explorations of the vulnerabilities of DRL by studying other aspects of attacks on DRL using realistic and efficient attacks. First, we adapt and propose efficient black-box attacks when we do not have access to DRL model parameters. Second, to address the high computational demands of existing attacks, we introduce efficient online sequential attacks that exploit temporal consistency across consecutive steps. Third, we explore the possibility of an attacker perturbing other aspects in the DRL setting, such as the environment dynamics. Finally, to account for imperfections in how an attacker would inject perturbations in the physical world, we devise a method for generating a robust physical perturbations to be printed. The attack is evaluated on a real-world robot under various conditions. We conduct extensive experiments both in simulation such as Atari games, robotics and autonomous driving, and on real-world robotics, to compare the effectiveness of the proposed attacks with baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.

Automated intersection delay estimation using the input–output principle and turning movement data

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

This paper investigates stock portfolios by application of Deep Reinforcement Learning (DRL) Models to achieve an optimal tactical asset allocation. The research problem is described as an optimization scenario that seeks to maximize the portfolio risk adjusted returns for a given portfolio asset allocation. The problem is set up with an initial capital investment which is invested in a set of assets. The initial strategic allocation is determined, which in our case is the equal weight allocation, and all of the capital is invested in the set of assets. At each point in time, the assets are reallocated according to the allocation which will increase the portfolio value. Two DRL models are implemented. The performance of the DRL models is compared with the uniform weights portfolio. The results show that, generally, two DRL models have higher cumulative returns.

Deep reinforcement learning based intelligent traffic light is an increasing tendency for improving the traffic condition due to its promising control performance. The current works about intelligent traffic lights tend to control the signal phase or phase time under fixed-lane condition. However, the fixed-lane condition cannot fit the different vehicle flow very well, thus the traffic condition cannot be enhanced further. In this paper, for improving the traffic condition further, we extend the existing intelligent traffic lights to the two-dimension control problem where the agent needs to control not only the traffic light signal, but also the direction of changeable lanes. As a result, the traffic light signal and lane allocation satisfy the need of current vehicle flow simultaneously. Furthermore, for the joint problem of lane allocation and the control of traffic light control, we develop a Q-network structure to build its deep reinforcement learning model. The simulation results demonstrate that the performance of the proposed method outperforms that of the existing intelligent traffic light with fixed lane.

Study on deep reinforcement learning techniques for building energy consumption forecasting

Irrigation decision-making using Reinforcement Learning (RL) performs well in changing environment, but easily falls into sub-optimal solutions with high-dimensional data. Deep Reinforcement Learning (DRL) has fused RL with Deep Learning (DL) and excels at learning adaptive and long-term irrigation strategies directly from high-dimensional environment data. This paper systematically reviews the applications of DRL in irrigation optimization, covering both pre-trained environments based on crop growth simulators and dynamic environments driven by real-time sensors. We discussed the strengths of classic DRL algorithms, including their ability to handle dynamic and non-linear environments, and reviewed their performance in irrigation multi-objective optimization and decision-making. In addition, we identified constraints in applying DRL in irrigation decision making, which include data scarcity, poor model interpretability, and difficulties in field deployment. It shows DRL can provide a powerful framework for adaptive irrigation, but is constrained by the gap between simulation and real-world complexity. To address these limitations, we discussed approaches in future work, such as developing multi-objective DRL algorithms. These approaches will improve DRL modeling outcomes and provide a technological foundation for smart agriculture and sustainable resource management. • Review the application of Deep Reinforcement Learning (DRL) in agricultural irrigation. • Analyze the performance of DRL algorithms in irrigation decision-making. • Compare DRL models based on different environment in irrigation optimization. • Discuss the further work to improve DRL performance in irrigation optimization.

Efficient congestion control in communications using novel weighted ensemble deep reinforcement learning

Traffic signal control is a complex problem and it is difficult to determine an optimal strategy to control multi-directional traffic at multiple intersections. Recent years have witnessed numerous successes of deep learning neural networks in the fields of artificial intelligence. Motivated by the dominant performance of neural networks, this study attempts to develop a novel adaptive signal control approach by fusing deep learning (DL) and reinforcement learning (RL), i.e., deep reinforcement learning (DRL), for arterial signal coordination. DRL can considerably improve the ability to deal with large amounts of data processing, systematic perception and expression, which is key to coordinated control of arterial intersections. The proposed algorithm is implemented by utilizing real-time traffic detection data and aims to optimize the hybrid global and local reward functions. The experimental results obtained by traffic simulation software SUMO demonstrate the advantage of the proposed approach, as well as its efficiency and effectiveness compared with fixed-time and actual signal control methods.

Intrusion detection has been a vast-surveyed topic for many decades as network attacks are tremendously growing. This has heightened the need for security in networks as web-based communication systems are advanced nowadays. The proposed work introduces an intelligent semi-supervised intrusion detection system based on different algorithms to classify the network attacks accurately. Initially, the pre-processing is accomplished using null value dropping and standard scaler normalization. After pre-processing, an enhanced Deep Reinforcement Learning (EDRL) model is employed to extract high-level representations and learn complex patterns from data by means of interaction with the environment. The enhancement of deep reinforcement learning is made by associating a deep autoencoder (AE) and an improved flamingo search algorithm (IFSA) to approximate the Q-function and optimal policy selection. After feature representations, a support vector machine (SVM) classifier, which discriminates the input into normal and attack instances, is employed for classification. The presented model is simulated in the Python platform and evaluated using the UNSW-NB15, CICIDS2017, and NSL-KDD datasets. The overall classification accuracy is 99.6%, 99.93%, and 99.42% using UNSW-NB15, CICIDS2017, and NSL-KDD datasets, which is higher than the existing detection frameworks.

Traffic congestion and delays at signalized intersections remain a challenge in urban mobility. To minimize congestion, adaptive traffic signal control systems have been adopted across the United States. However, for these systems to function at their best, systems require an accurate estimation of traffic states, such as traffic density and queue length. This paper implements a relatively novel method, Physics-Informed Neural Networks (PINNs), for real-time traffic density estimation to enhance adaptive intersection signal control. The methodology leverages PINNs, which integrate both data-driven learning and physical constraints imposed by the Lighthill-Whitham-Richards (LWR) macroscopic traffic flow model. A simulation of a four-way intersection was used to generate realistic spatio-temporal traffic data to evaluate the performance of the model. Separate PINNs models, one trained for each incoming approach using simulation-based spatio-temporal density data, provide segment-level density predictions. These real-time density estimates are then employed within an adaptive traffic signal control algorithm designed to adjust green light phasing based on detected demand. The proposed PINNs control strategy was evaluated through simulation, comparing its performance against the Webster traffic timing method using key metrics such as average vehicle delay at the intersection and average vehicle travel time. The findings show an approximate 18% decrease in average vehicle delay and more than a 6% reduction in average travel time when utilizing adaptive traffic control based on the PINNs method for determining density values. These findings highlight the potential of leveraging PINNs for accurate, real-time traffic state estimation, which can effectively inform and improve the performance of adaptive traffic signal control systems, leading to reduced congestion and enhanced mobility at intersections.

To reduce travel time, the actuated signal controls have been implemented at urban intersections. However, the safety impacts of actuated signal controls thus far have rarely been examined. In this assessment of the safety impact of urban intersections with semi-actuated signal controls, the safety performance functions and EB approaches were applied. The semi-actuated signal controls have increased injuries and total crashes in all crash types by around 5.9% and 3.8%, respectively. Regarding the most common crash types, such as angle, sideswipe & rear-end, and head-on crashes, semi-actuated signal controls have been seen to decrease injuries by 7.7%. Total crashes have been reduced by over 9.2% through the use of semi-actuated signal controls. This may be result of optimal signal timings considering traffic conditions during peak time periods. In conclusion, safety impact factors which have been established in this study can be used to improve safety and minimize travel times using semi-actuated signal controls.

Adaptive traffic signal controllers offer better signal time management especially when the traffic flow pattern is not uniform on all approaches. Traditional adaptive traffic controller use upstream or advance vehicle detection which works well in situations where traffic follows good lane discipline. However, when the spacing between intersections increases or in the case of complex geometry these systems may not be efficient. This is primarily because of the inability of traffic flow models to accurately estimate the traffic demand from the upstream detectors. Using stop-line detector information is best suited in such traffic conditions as they do not require any explicit prediction models. Furthermore, there are many intersections which works using stop-line detectors with preset maximum green timings as vehicle actuated controllers. These controllers can be easily converted into truly adaptive by changing their maximum green timings continuously with respect to changing traffic flow pattern. Hence, this paper proposes an adaptive traffic control model which uses stop-line detector information instead of upstream detector. The model aims at real-time allocation of green time through reinforcement learning; an approach originated from the machine learning community. This approach has the ability to learn relationships between signal control actions and their effect on the queue while pursuing the goal of maximizing throughput which is a distinct improvement over the traditional vehicle actuated system. To demonstrate the performance of the proposed model a typical four-way intersection with four-phase scheme is evaluated for various flow conditions with the proposed model as well as with the traditional vehicle actuated system. The results show improvement over traditional system, especially when the flow is near the capacity.