Simulation of Smart Traffic Light By Using Image Processing and Reinforcement Learning

The Indian traffic control system faces lots of difficulties due to the increasing volume of vehicles, ineffective systems for traffic administration during peak hours, and the frequent need for manual intervention due to the inadequate performance of traffic signals in managing heavy traffic flow. Traditional traffic lights in India have defined timings for each lane, which frequently cause longer traffic jams in lanes with more traffic. This study presents an intelligent traffic control system that incorporates the YOLOv9 model for real-time traffic length prediction and intelligently allocates green, red, and orange signal timings. YOLOv9 builds a bounding box that allows it to compute vehicle density precisely by enclosing the initial and final cars in every frame. As a comparison with traditional fixed-time methods, the proposed approach recalculates traffic signal timings at each cycle based on the recorded length of traffic. In order to ensure that emergency services respond more quickly, the system efficiently prioritizes emergency vehicles by quickly moving them from their lane when they are spotted. This adaptive strategy aligns signal duration with real traffic demand, boosting overall traffic efficiency and regulating traffic flow by reducing unnecessary wait times for low-traffic lanes. In comparison to fixed-timing systems and object detection strategies, our research on adaptive traffic systems reveals a 66% to 77% decrease in vehicle delay. Compared to traditional fixed-timing approaches, the proposed method demonstrates significant enhancements in effectively managing traffic congestion.

Traffic lights at crossroads help regulate the flow of vehicles so they can run smoothly and avoid congestion. One of the causes of traffic congestion is a traffic control light system that is not optimal because it still uses a fixed timer system, so the lights' duration is less efficient in its application. When the green light is on, the streets are deserted. But there is a traffic jam on another road. Therefore the concept of traffic light timing needs to be developed to get efficient timing. This study aims to create a traffic light control system that uses a PIR (Passive Infrared) motion sensor and an ultrasonic sensor based on the Arduino Mega2560 microcontroller based on the congestion level. The method used in this traffic control prototype is an experimental method of reading vehicle objects mounted with PIR sensors and ultrasonic sensors. This system can detect the movement of vehicles approaching a traffic light, measure the distance between the car and the traffic light, and set the timing of the light signal according to the level of congestion detected. This system has been tested using vehicle movement simulations brought closer to the PIR and ultrasonic sensors at different distances. The test results show the system can work properly and produce the light signal according to the detected traffic conditions. For its application to traffic lights in the future, it is feasible to help the traffic unit.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

Reinforcement learning (RL) models have been broadly used in modeling the choice behavior of humans and other animals. In standard RL models, the action values are assumed to be updated according to the reward prediction error (RPE), i.e., the difference between the obtained reward and the expected reward. Numerous studies have noted that the magnitude of the update is biased depending on the sign of the RPE. The bias is represented in RL models by differential learning rates for positive and negative RPEs. However, which aspect of behavioral data that the estimated differential learning rates reflect is not well understood. In this study, we investigate how the differential learning rates influence the statistical properties of choice behavior (i.e., the relation between past experiences and the current choice) based on theoretical considerations and numerical simulations. We clarify that when the learning rates differ, the impact of a past outcome depends on the subsequent outcomes, in contrast to standard RL models with symmetric value updates. Based on the results, we propose a model-neutral statistical test to validate the hypothesis that value updates are asymmetric. The asymmetry in the value updates induces the autocorrelation of choice (i.e., the tendency to repeat the same choice or to switch the choice irrespective of past rewards). Conversely, if an RL model without an intrinsic autocorrelation factor is fitted to data that possess an intrinsic autocorrelation, a statistical bias to overestimate the difference in learning rates arises. We demonstrate that this bias can cause a statistical artifact in RL-model fitting leading to a “pseudo-positivity bias” and a “pseudo-confirmation bias.”

The relation between reinforcement learning parameters and the influence of reinforcement history on choice behavior

Experimental research such as cell cultures and carbon nanotube (CNT) growth are largely governed by following predefined execution protocols with fine-tuned control of parameters. There are promising opportunities to apply reinforcement learning (RL), an established learning technique in the area of artificial intelligence, in order to automate CNT growth process and accelerate related scientific breakthroughs in material discovery. Although there are benefits in RL-based exploration and exploitation methodologies, there are also challenges in developing relevant learning policies in experimental settings relating to CNT growth. In this paper, we present a novel data-driven RL approach for assisting experimental CNT growth. Our approach focuses on developing an RL model to learn from simulation-based images and characteristics of temporal CNT growth, considering various growth parameters. Our RL model learns from CNT growth variation in a simulation-based environment where critical control parameters, such as density, growth rate, tube radius, tube stiffness, and Van der Waals forces, are used. Our RL model enables CNT growth automation in order to explore of a wider range of growth conditions, and improve reproducibility. The ultimate goal of our RL model is to achieve desired CNT growth by dynamically controlling growth parameters throughout a sequence of experiments. We evaluate the effectiveness of our RL approach by measuring the improvement in the maximum compressive strength of a carbon nanotube ‘with’ and ‘without’ the RL model. Our results show the effectiveness of course correction recommended by our RL approach when controlling the parameters of angular deviation and rate of growth of carbon nanotubes, when compared against non-regulated CNT growth.

Road congestion is a huge problem that people face most of the time, especially those who live in encumbered cities, which wastes time and money. Mainly, traditional traffic light control systems are far from providing effective urban traffic management in terms of journey time and waiting time and are unable to keep pace with the development of smart cities and urban mobility requirements. This article proposes an intelligent control algorithm of traffic lights. It is based on two important strategies, the first of which is the calculation of the shortest path to be performed between a source and a destination. The second strategy is an algorithm for managing and controlling traffic lights according to the number of vehicles on the road network studied. The number of vehicles is obtained using sensors placed on each road. Later, the number of vehicles is used to calculate the shortest path by applying an improved version of the Dijkstra algorithm. There is a controller who communicates with their neighbors in order to collect the necessary data and control the operation of traffic lights in an intersection. All the algorithms, including the calculation of the shortest path and the synchronization of the traffic lights are validated by traffic simulators that have been implemented using java language. The results show that the traffic light timing is calculated successfully.

Fog computing extends cloud services to the network edge, enabling low-latency processing for Internet of Things (IoT) applications. However, this distributed approach is vulnerable to a wide range of attacks, necessitating advanced intrusion detection systems (IDSs) that operate under resource constraints. This study proposes integrating self-awareness (online learning and concept drift adaptation) into a lightweight RL (reinforcement learning)-based IDS for fog networks and quantitatively comparing it with non-RL static thresholds and bandit-based approaches in real time. Novel self-aware reinforcement learning (RL) models, the Hierarchical Adaptive Thompson Sampling–Reinforcement Learning (HATS-RL) model, and the Federated Hierarchical Adaptive Thompson Sampling–Reinforcement Learning (F-HATS-RL), were proposed for real-time intrusion detection in a fog network. These self-aware RL policies integrated online uncertainty estimation and concept-drift detection to adapt to evolving attacks. The RL models were benchmarked against the static threshold (ST) model and a widely adopted linear bandit (Linear Upper Confidence Bound/LinUCB). A realistic fog network simulator with heterogeneous nodes and streaming traffic, including multi-type attack bursts and gradual concept drift, was established. The models’ detection performance was compared using metrics including latency, energy consumption, detection accuracy, and the area under the precision–recall curve (AUPR) and the area under the receiver operating characteristic curve (AUROC). Notably, the federated self-aware agent (F-HATS-RL) achieved the best AUROC (0.933) and AUPR (0.857), with a latency of 0.27 ms and the lowest energy consumption of 0.0137 mJ, indicating its ability to detect intrusions in fog networks with minimal energy. The findings suggest that self-aware RL agents can detect traffic–dynamic attack methods and adapt accordingly, resulting in more stable long-term performance. By contrast, a static model’s accuracy degrades under drift.

Reinforcement learning (RL) models have been influential in characterizing human learning and decision making, but few studies apply them to characterizing human spatial navigation and even fewer systematically compare RL models under different navigation requirements. Because RL can characterize one’s learning strategies quantitatively and in a continuous manner, and one’s consistency of using such strategies, it can provide a novel and important perspective for understanding the marked individual differences in human navigation and disentangle navigation strategies from navigation performance. One-hundred and fourteen participants completed wayfinding tasks in a virtual environment where different phases manipulated navigation requirements. We compared performance of five RL models (3 model-free, 1 model-based and 1 “hybrid”) at fitting navigation behaviors in different phases. Supporting implications from prior literature, the hybrid model provided the best fit regardless of navigation requirements, suggesting the majority of participants rely on a blend of model-free (route-following) and model-based (cognitive mapping) learning in such navigation scenarios. Furthermore, consistent with a key prediction, there was a correlation in the hybrid model between the weight on model-based learning (i.e., navigation strategy) and the navigator’s exploration vs. exploitation tendency (i.e., consistency of using such navigation strategy), which was modulated by navigation task requirements. Together, we not only show how computational findings from RL align with the spatial navigation literature, but also reveal how the relationship between navigation strategy and a person’s consistency using such strategies changes as navigation requirements change.

Deep reinforcement learning based intelligent traffic light is an increasing tendency for improving the traffic condition due to its promising control performance. The current works about intelligent traffic lights tend to control the signal phase or phase time under fixed-lane condition. However, the fixed-lane condition cannot fit the different vehicle flow very well, thus the traffic condition cannot be enhanced further. In this paper, for improving the traffic condition further, we extend the existing intelligent traffic lights to the two-dimension control problem where the agent needs to control not only the traffic light signal, but also the direction of changeable lanes. As a result, the traffic light signal and lane allocation satisfy the need of current vehicle flow simultaneously. Furthermore, for the joint problem of lane allocation and the control of traffic light control, we develop a Q-network structure to build its deep reinforcement learning model. The simulation results demonstrate that the performance of the proposed method outperforms that of the existing intelligent traffic light with fixed lane.

Data quality is a critical aspect of data analytics since it directly influences the accuracy and effectiveness of insights and predictions generated from data. Artificial Intelligence (AI) schemes have grown in the existing era of technological advancement, which provides innovative exposure to healthcare applications. Reinforcement Learning (RL) is a subfield and an influential Machine Learning (ML) model aimed at optimizing decision-making by association with dynamic environments. In healthcare applications, RL can modify conduct strategies, enhance source application, and improve patient investigation history by using various data modalities. The worth of the data quality regulates how effective RL is in healthcare applications. In healthcare, the model predictions have a direct impact on patient's lives, and poor data quality often leads to wrong evaluations that expose patient safety and treatment quality. Biases in data quality have also presented a challenging influence on the RL model's effectiveness and accuracy. RL models have enormous potential in healthcare; however, various strategic limitations prevent their widespread acceptance and deployment. The implementation of RL in healthcare faces serious issues, mostly around data quality, bias, and tactical difficulties. This study delivers a broad survey of these challenges, emphasizing how imbalanced, imperfect, and biased data can affect the generalizability and performance of RL models. We critically assessed the sources of data bias, comprising demographic imbalances and irregularities in electronic health records (EHRs), and their impact on RL algorithms. This survey aims to present a detailed study of the complex circumstances relating to data quality, data biases, and strategic barriers in RL models deploying in healthcare applications. However, the main contribution of the proposed study is that it provides a systematic review of these challenges and delivers a roadmap for future work intended to refine the consistency, fairness, and scalability of RL in healthcare sectors.

This paper is aimed to describe the structural complexity of the traffic light control system. The structural complexity represents an interrelationship indicator between the various elements functions. The traffic light control system has several additional features that are synchronized to the railway doorstop, and the time interval for the green signals is expanded. For determining the method of its structural complexity, the Petri net model is used. Three kinds of traffic lights models are explored, namely standard, Norwegian, and the Norwegian improvement. The study results indicate that the Norwegian traffic lights have the most complex structure when regular implementation. The runner up is the Norwegian improvement. The most ideal is the standard traffic lights. Norwegian improvement is the most ideal when integrated with the railway doorstop and implemented extra green signals time interval. The Norwegian traffic light is the second. The standard traffic lights structural complexity fluctuates. It means that the Norwegian traffic light and its improvement are suitably used. Adding several features improve the structural complexity approaching the ideal system while extra controllers accompany each element.
 HIGHLIGHTS
 
 The Petri net model can represent the structural complexity of a system
 The method that can measure the structural complexity becomes ideal for actual implementation
 The Norwegian traffic lights structural complexities and improvements are ideal for the systems synchronized to the railway and appropriate for systems using the extra green signal time intervals or other additional features
 
 GRAPHICAL ABSTRACT

Rapid urbanization and increasing ownership of private vehicles have led to severe traffic congestion, which has become a critical challenge in modern urban areas operated with signalized intersections. This study proposes and evaluates a novel AI-based composite framework for traffic signal control optimization, which integrates a Long Short-Term Memory (LSTM) model for traffic flow prediction, a Deep Neural Network (DNN) for vehicle speed estimation, and a Genetic Algorithm (GA) for signal phase optimization under open-loop and closed-loop objectives. It has been validated through high-fidelity microscopic simulations (i.e. VISSIM), calibrated with field data at the atypical six-leg intersection in Busan, Korea. This study found that while the overall performance of the framework for 24 hours exhibits clear trade-offs (i.e. slight total traffic throughput decrease of 2.2%), its primary value lies in its demonstrated ability to effectively mitigate severe congestion during peak hours (i.e. speed increase of up to 38.5% during the evening peak), delivering significant gains in traffic quality in need. This study also revealed that the system actively performs traffic flow redistribution as an intelligent traffic control strategy. The contributions of this study are threefold. First, it provides a methodological blueprint for an interpretable, modular control framework that decouples prediction processes from optimization, offering a transparent alternative to end-to-end Reinforcement Learning (RL) models. Second, this study offers crucial empirical insight for the evaluation of adaptive traffic control systems. Their true effectiveness may be best measured by peak hour performance rather than 24-hour averages. Finally, this study makes a conceptual contribution by identifying and quantifying the ‘reality gap’ between the predictions of an offline-trained optimizer and the outcomes of a high-fidelity simulation, highlighting a fundamental challenge for the field. HIGHLIGHTS A hierarchical AI framework is an alternative to reinforcement learning model Closed-loop outperforms open-loop in optimizing real time traffic signal control A speed stabilization objective is adopted over a speed maximization approach Leading to tangible gains in traffic quality during peak hour over 24-hour average

Stress and genetic background regulate different aspects of behavioral learning through the action of stress hormones and neuromodulators. In reinforcement learning (RL) models, meta-parameters such as learning rate, future reward discount factor, and exploitation-exploration factor, control learning dynamics and performance. They are hypothesized to be related to neuromodulatory levels in the brain. We found that many aspects of animal learning and performance can be described by simple RL models using dynamic control of the meta-parameters. To study the effects of stress and genotype, we carried out 5-hole-box light conditioning and Morris water maze experiments with C57BL/6 and DBA/2 mouse strains. The animals were exposed to different kinds of stress to evaluate its effects on immediate performance as well as on long-term memory. Then, we used RL models to simulate their behavior. For each experimental session, we estimated a set of model meta-parameters that produced the best fit between the model and the animal performance. The dynamics of several estimated meta-parameters were qualitatively similar for the two simulated experiments, and with statistically significant differences between different genetic strains and stress conditions.

Metamaterials are engineered composites that can exhibit acoustic, electromagnetic, elasto-dynamic and mechanical properties that are not found in natural materials Due to the complexity of the target objective functions, it is difficult to find the globally optimized solutions in the inverse design of metameterials. This thesis proposes and outlines two model, a gradient-based optimization method combined with generative networks (2D-GLOnets) and a reinforcement learning (RL) model, that can find the optimized metamaterial structures across a wide range of parameters. By perturbing the positions of each cylindrical scatterer in a planar configuration, 2D-GLOnets and the RL model with Deep Deterministic Policy Gradients (DDPG) are developed to design 2D (two-dimensional) broadband acoustic cloaking devices at the desired range of wavenumbers. Both models were implemented using PyTorch, Python libraries, and MATLAB engine. The numerical results are presented and compared with the optimal values produced by fmincon algorithms to verify the validation of our approaches. Our results indicated that both methods reduced the total scattering cross section by 17% to 90% compared to the initial conditions.