Flow-Through Tensors: A Unified Computational Graph Architecture for Multi-Layer Transportation Network Optimization

Multi entity perspective freight demand modeling technique: Varying objectives and outcomes

This paper addressed the vehicle routing problem (VRP) in large-scale urban transportation networks with stochastic time-dependent (STD) travel times. The subproblem which is how to find the optimal path connecting any pair of customer nodes in a STD network was solved through a robust approach without requiring the probability distributions of link travel times. Based on that, the proposed STD-VRP model can be converted into solving a normal time-dependent VRP (TD-VRP), and algorithms for such TD-VRPs can also be introduced to obtain the solution. Numerical experiments were conducted to address STD-VRPTW of practical sizes on a real world urban network, demonstrated here on the road network of Shenzhen, China. The stochastic time-dependent link travel times of the network were calibrated by historical floating car data. A route construction algorithm was applied to solve the STD problem in 4 delivery scenarios efficiently. The computational results showed that the proposed STD-VRPTW model can improve the level of customer service by satisfying the time-window constraint under any circumstances. The improvement can be very significant especially for large-scale network delivery tasks with no more increase in cost and environmental impacts.

Flow-through tensors: A unified computational graph architecture for multi-layer transportation network optimization

A weighted edge-coloured graph is a graph for which each edge is assigned both a positive weight and a discrete colour, and can be used to model transportation and computer networks in which there are multiple transportation modes. In such a graph paths are compared by their total weight in each colour, resulting in a Pareto set of minimal paths from one vertex to another. This paper will give a tight upper bound on the cardinality of a minimal set of paths for any weighted edge-coloured graph. Additionally, a bound is presented on the expected number of minimal paths in weighted edge–bicoloured graphs. These bounds indicate that despite weighted edge-coloured graphs are theoretically intractable, amenability to computation is typically found in practice.

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, the centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. The multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now, the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent, advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. The results demonstrate its optimality, robustness, and sample efficiency over the other state-of-the-art decentralized MARL algorithms.

This paper has proposed a user-density-based coverage optimization technique for ultra-dense cellular networks. Antenna tilting is a promising coverage optimization technique to be used in 5G networks, that significantly improve the signal to interference plus noise ratio (SINR) by choosing the appropriate angle of tilt. In this paper, the cellular coverage has been optimized for scattered user densities/user hotspots using an adaptive antenna tilting mechanism that steers the beams towards the temporal hot spot in the coverage area. The proposed method has the competence to improve the desired SINR level and coverage area for a group of users rather than a single user. In this work, a reinforcement learning (RL) algorithm has been implemented to optimize the tilt angle. The performance of the proposed technique has been evaluated in the simulation platform considering a three-sectored multicellular mobile network where the groups of user clusters are distributed randomly. The result confirms the improvement in RSS and SINR values in the group of users having high density with maximum user satisfaction.

Designing well-functioning and fair transport networks is not a trivial task, given the large space of solutions and constraints one must satisfy. Moreover, different spatial segregation sources can render some transportation network interventions unfair to specific groups. It is thereby crucial to optimize the transportation system while mitigating the disproportional benefits it can lead to. In this paper, we explore the trade-off between efficiency and fairness in the Transport Network Design Problem (TNDP), via the use of Deep Reinforcement Learning (Deep RL). We formulate different fairness definitions as reward functions - inspired by Equal Sharing of Benefits, Narrowing the Gap, and Rawl's justice theory. We apply our method to Amsterdam (The Netherlands) and Xi'an (China) and show that vanilla Deep RL can lead to biased outcomes. By considering different fair rewards, however, we can shed light on possible compromises between fairness and efficiency in the TNDP.

Reinforcement learning (RL) approaches have recently been spotlighted for use in adaptive traffic-signal control on an area-wide level. Most researchers have employed multi-agent reinforcement learning (MARL) algorithms wherein each agent shares a holistic traffic state and cooperates with other agents to reach a common goal. However, MARL algorithms cannot guarantee a global optimal solution unless the actions of all agents are fully coordinated. The present study employs a RL algorithm that recognizes an entire traffic state and jointly controls all the traffic signals of multiple intersections. With this approach, a deep Q-network (DQN) that depends solely on traffic images is extended to overcome the curse of dimensionality that is associated with a large state and action space. Several front layers in a deep convolutional neural network (CNN) to approximate the true Q-function are shared by each intersection approach. Weight parameters connecting the last hidden layer to the output layer are fixed. The proposed methodology outperforms a fixed-signal operation, a fully actuated signal operation, a multi-agent RL control without coordination, and a multi-agent RL control with partial coordination.

The recent progress in Machine Learning (Géron, 2022) and particularly Deep Learning (Goodfellow, 2016) models exposed the limitations of traditional computer architectures. Modern algorithms demonstrate highly increased computational demands and data requirements that most existing architectures cannot handle efficiently. These demands result in training speed, inference latency, and power consumption bottlenecks, which is why advanced methods of computer architecture optimization are required to enable the development of ML/DL-dedicated efficient hardware platforms (Engineers, 2019). The optimization of computer architecture for applications of ML/DL becomes critical, due to the tremendous demand for efficient execution of complex computations by Neural Networks (Goodfellow, 2016). This paper reviewed the numerous approaches and methods utilized to optimize computer architecture for ML/DL workloads. The following sections contain substantial discussion concerning the hardware-level optimizations, enhancements of traditional software frameworks and their unique versions, and innovative explorations of architectures. In particular, we discussed hardware including specialized accelerators, which can improve the performance and efficiency of a computation system using various techniques, specifically describing accelerators like CPUs (multicore) (Hennessy, 2017), GPUs (Hwu, 2015) and TPUs (Contributors, 2017), parallelism in multicore architectures, data movement in hardware systems, especially techniques such as caching and sparsity, compression, and quantization, other special techniques and configurations, such as using specialized data formats, and measurement sparsity. Moreover, this paper provided a comprehensive analysis of current trends in software frameworks, Data Movement optimization strategies (A.Bienz, 2021), sparsity, quantization and compression methods, using ML for architecture exploration, and, DVFS (Hennessy, 2017),, which provides strategies for maximizing hardware utilization and power consumption during training, machine learning, dynamic voltage, and frequency scaling, runtime systems. Finally, the paper discussed research opportunity directions and the possibilities of computer architecture optimization influence in various industrial and academic areas of ML/DL technologies. The objective of implementing these optimization techniques is to largely minimize the current gap between the computational needs of ML/DL algorithms and the current hardware’s capability. This will lead to significant improvements in training times, enable real-time inference for various applications, and ultimately unlock the full potential of cutting-edge machine learning algorithms.

The success of autonomous mobile robots depends on the ability to understand continuously changing scenery. Present techniques for analysis of images are not always suitable because in sequential paradigm, computation of visual functions based on absolute values of stimuli is inefficient. Important aspects of visual information are encoded in discontinuities of intensity, hence a representation in terms of relative values seems advantageous. We present the computing architecture of a massively parallel vision module which optimizes the detection of relative intensity changes in space and time. Visual information must remain constant despite variation in ambient light level or velocity of target and robot. Constancy can be achieved by normalizing motion and lightness scales. In both cases, basic computation involves a comparison of the center pixels with the context of surrounding values. Therefore, a similar computing architecture, composed of three functionally-different and hierarchically-arranged layers of overlapping operators, can be used for two integrated parts of the module. The first part maintains high sensitivity to spatial changes by reducing noise and normalizing the lightness scale. The result is used by the second part to maintain high sensitivity to temporal discontinuities and to compute relative motion information. Simulation results show that response of the module is proportional to contrast of the stimulus and remains constant over the whole domain of intensity. It is also proportional to velocity of motion limited to any small portion of the visual field. Uniform motion throughout the visual field results in constant response, independent of velocity. Spatial and temporal intensity changes are enhanced because computationally, the module resembles the behavior of a DOG function.

Autonomous digital twin framework for gas turbine combined cycle control loops: Comparative study of proportional-integral control, reinforcement learning, and reinforcement learning with agents

This paper studies the challenge of efficient resource allocation in wireless networks by investigating the potential benefits of cognitive radio networks (CRN) and non‐orthogonal multiple access (NOMA) techniques. Specifically, the purpose of this paper is to explore user pairing and power allocation in a multiuser cooperative CRN‐NOMA network. To achieve the goal, a novel user pairing algorithm with reinforcement learning (RL) is proposed to adaptively select the optimal group for pairing according to the factors such as channel gain, distances, and angles. A user group is consisted of a primary user (PU) and up to three secondary users (SUs). The key findings of this paper highlight the effectiveness of RL in optimizing power allocation while considering various constraints. As a result, significant improvements in network throughput and user fairness are achieved. Compared with the classical power allocation algorithms (i.e., equal power allocation, proportional power allocation, iterative power allocation, and iterative water‐filling power allocation), the proposed power allocation based on RL algorithm and iterative water‐filling power allocation algorithm have better performance in terms of throughput, fairness, spectrum efficiency, and energy efficiency. In addition, compared with the iterative water‐filling power algorithm, the proposed algorithm improves the throughput by 31.71% and obtains twice the spectrum efficiency and energy efficiency. The simulation results consistently show that the proposed method has advantages in enhancing PU capacity. Further advancements in this field of research are encouraged based on the promising outcomes of this study.

This paper introduces a groundbreaking reinforcement learning (RL)-driven optimization framework for cyclic redundancy check (CRC) polynomials, tailored to meet the ultra-reliable low-latency communication (URLLC) and massive machine-type communication (mMTC) demands of 5G and beyond wireless networks. By addressing the critical challenge of undetectable error patterns (UEPs) in guessing random additive noise decoding (GRAND), our approach significantly enhances error detection and correction capabilities under stringent reliability constraints. Leveraging a markov decision process (MDP) model, we optimize CRC polynomial coefficients through Q-learning with -greedy exploration, where each action flips a coefficient and rewards are based on UEP reduction. This method ensures robust performance in additive white Gaussian noise (AWGN) channels, aligning with the ultra-reliability requirements of URLLC and the scalability needs of mMTC. Experimental results demonstrate substantial improvements in bit error rate (BER) and decoding reliability, paving the way for next-generation wireless standards. Our work bridges the gap between theoretical innovation and practical deployment, offering a scalable, learning-based solution for future high-performance communication systems.

Advances in deep multi-agent reinforcement learning learning(MARL) enable sequential decision making for a range of exciting multi-agent applications. The black-box characteristic of MARL restricts the safe and scalable application of decision models in practical deployment. However, existing interpretability methods for deep reinforcement learning models are not suitable for addressing challenges posed by multi-agent environments and often inadequate in generating logical sequential decisions. We present an innovative framework called BT4MARL, which introduces the behavior tree structure to explainable MARL. The proposed method clusters state space by aggregating temporally related states and divides agents into several groups in the new state. Based on these clustered states and agents, we constructs behavior tree structures. In this way, we use an exploration technique based on pairing a combined behavior tree with the target model. We empirically show that our framework is effective in four benchmark MARL domains. Moreover, the results of a user study show that the generated explanations significantly improve performance and satisfaction. This work represents a significant stride towards addressing the challenges of explainability and performance in MARL applications.