Deep Reinforcement Learning Strategies for Cloud-Edge Multipath Transmission in Cloud Computing Data Services

Space-ground integrated networks (SGINs) have been recently regarded as a promising way to provide resilient, dependable as well as efficient data transmission in the high-speed railway (HSR) scenario. Applying multipath transmission control protocol (MPTCP) to SGIN can realize data transmission simultaneously via terrestrial and satellite networks. However, since the existing congestion control (CC) mechanisms of MPTCP fail to distinguish between adverse influences (such as packet loss and/or round-trip time increase) caused by congestion and those caused by handovers, it suffers severe performance degradation in the HSR scenario where handover frequently occurs. In this article, we first present the SGIN oriented HSR (SGIN-HSR) with MPTCP. Then leveraging cross-layer information (i.e., reference signal received power), we design a novel cross-layer aided MPTCP CC mechanism targeted at SGIN-HSR based on deep reinforcement learning, which is referred to as HSR-CC, to alleviate performance degradation problems induced by handover. The experimental results show that HSR-CC significantly enhances the goodput and outperforms state-of-the-art MPTCP CC algorithms in SGIN-HSR environments where handover frequently occurs.

With the growing demand of Internet services, network operators have put significant efforts to improve network error resilience and efficiency. Since there exist different wired/wireless technologies for Internet access such as digital subscriber line (DSL), Ethernet, and worldwide interoperability for microwave access (WiMax), a mobile host can use multiple access networks simultaneously with multipath transmission. Taking the advantage of heterogeneous environment, multipath transmission through the Internet can improve service reliability and network flexibility. Ensuring a reliable end-to-end connection oriented communication with satisfactory quality of service (QoS) and maintaining congestion control and flow control are the main responsibilities of the Transmission Control Protocol (TCP), the dominant transport layer protocol in the Internet. In this paper, we survey the state-of-the-art of multipath transmission techniques for QoS provisioning in wireless Internet access, focusing on the end-to-end transport layer protocols. The main challenges for the design of multipath TCP are reviewed, and the existing transport layer congestion control schemes are categorized. Multipath TCP and stream control transmission protocol (SCTP)-based transport layer protocols are discussed, and their limitations and/or impractical assumptions are addressed. Open research issues on the development of an effective, efficient, and practical multipath TCP protocol are summarized.

Multipath Transmission Control Protocol (MPTCP) is a new transmission protocol designed by The Internet Engineering Task Force (IETF) for multipath transmission. In order to realize the large-scale and rapid deployment of MPTCP, the design of MPTCP needs to ensure fairness with traditional Transmission Control Protocol (TCP). The IETF organization designed a congestion control algorithm for the network fairness of MPTCP, but did not consider the bottleneck fairness of MPTCP. In this paper, we propose a congestion control algorithm based on Dynamic Time Warping (DTW) and Bottleneck Bandwidth and Round-trip time (BBR), which takes the bottleneck fairness in consideration for higher transmission efficiency. The experiments validate that the proposed algorithm can improve the recognition rate of shared bottlenecks, increase bandwidth utilization, and ensure fairness at the bottleneck.

Multipath Transmission Control Protocol (MPTCP) enables multi-homed devices to establish multiple simultaneous routes for data transmission. Congestion Control (CC) is a fundamental mechanism for implementing and designing MPTCP. The Internet of Things (IoT) networks generate a massive volume of heterogeneous traffic with high dimensional states and diverse QoS characteristics. The existing MPTCP CC algorithms are unable to perform efficiently under highly mobile and dynamic IoT environments. We propose a novel model-free SDN-based adaptive actor-critic deep reinforcement learning framework based on a fuzzy normalized neural network to address the issue of CC for MPTCP in the IoT networks. In the proposed method, an agent can learn efficiently and better approximate the state-action value function of the actor and the action function of the critic to adjust the sub-flows congestion windows size adaptively according to the dynamic condition of a network. Simulation results show that the proposed scheme outperforms the state of the art schemes in terms of the goodput under highly-dynamic IoT environments.

Multipath Transmission Control Protocol (MPTCP) is in a rapid development process. MPTCP allows a network node to utilize multiple network interfaces and IP paths at the same time. It can take full advantage of network resources and provide reliable transmission, which brings advantages to users in terms of performance and reliability. In order to study the performance of MPTCP under high load, this paper uses Mininet to create an SDN environment and compare the performance differences between MPTCP and TCP under high load, and simulates a common Web application network architecture. The performance of MPTCP under this architecture is tested and analyzed. The test results show that MPTCP performs better than traditional TCP under high load. Besides, its performance can be optimized furtherly by adjusting related parameters.

The spread of Edge Internet of Things (IoT) radically changes our lifestyle. However, the multimedia services in edge IoT are still stuck by inefficiency. The dynamic typologies perplex the transmission of massive real-time data. To solve this problem, multipath transmission control protocol (MPTCP) which has a natural advantage in transmission robustness and bandwidth aggregation is becoming a good choice. In this paper, failure-aware and delay-predicted multipath virtual queue scheduling (FD-MVQS) is proposed to optimize the MPTCP performance in edge IoT. FD-MVQS constructs a two-plane cooperative scheduling system. In the control plane, the transmission failure estimation and chaos theory-based arrival delay prediction methods are introduced to provide the foundation for prescheduling. In the data plane, the multipath virtual queue scheduling is designed to allocate segments to different subflows. Simulation results showed that the proposed FD-MVQS performed better than standard and typical multipath transmission solutions in throughput, delay, and segment disorder.

A path characteristics based adaptive subflow switching algorithm for dynamic heterogeneous MPTCP network

Mobile devices are able to leverage diverse heterogeneous network paths by Multi-Path Transmission Control Protocol (MPTCP); nevertheless, boosting MPTCP throughput in wireless networks is a real bear. Not only the best path(s) should be selected, but also the optimal congestion control mechanism should be chosen. We investigate the impact of different paths and congestion control for different signal quality states. Consequently, we present the novel MPTCP algorithm augmenting the end user throughput by understating the best policy in different situations by Q-learning. The Results reveal a tremendous effect of switching between the different interfaces and changing the congestion control mechanism on throughput and delay. By and large, the proposed framework achieves 10% more throughput compared to base MPTCP.

Adaptive traffic signal control (ATSC) based on deep reinforcement learning (DRL) has shown promising prospects to reduce traffic congestion. Most existing methods keeping traffic signal phases fixed adopt two agent actions to match a four-phase suffering unstable performance and undesirable operation in a four-phase signalized intersection. In this paper, a Double Deep Q-Network (DDQN) with a dual-agent algorithm is proposed to obtain a stable traffic signal control policy. Specifically, two agents are denoted by two different states and shift the control of green lights to make the phase sequence fixed and control process stable. State representations and reward functions are presented by improving the observability and reducing the leaning difficulty of two agents. To enhance the feasibility and reliability of two agents in the traffic control of the four-phase signalized intersection, a network structure incorporating DDQN is proposed to map states to rewards. Experiments under Simulation of Urban Mobility (SUMO) are carried out, and results show that the proposed traffic signal control algorithm is effective in improving traffic capacity.

Intelligent energy management for an on-grid hydrogen refueling station based on dueling double deep Q network algorithm with NoisyNet

Cyber-physical systems (CPSs) in autonomous vehicles must handle highly dynamic and uncertain settings, where unanticipated impediments, shifting traffic conditions, and environmental changes all provide substantial decision-making issues. Deep reinforcement learning (DRL) has emerged as a strong tool for dealing with such uncertainty, yet current DRL models struggle to ensure safety and optimal behaviour in indeterminate settings due to the difficulties of understanding dynamic reward systems. To address these constraints, this study incorporates double deep Q networks (DDQN) to improve the agent's adaptability under uncertain driving conditions. A structured reward system is established to accommodate real-time fluctuations, resulting in safer and more efficient decision-making. The study acknowledges the technological limitations of automobile CPSs and investigates hardware acceleration as a potential remedy in addition to algorithmic enhancements. Because of their post-manufacturing adaptability, parallel processing capabilities, and reconfigurability, field programmable gate arrays (FPGAs) are used to execute reinforcement learning in real-time. Using essential parameters, including collision rate, behaviour similarity, travel distance, speed control, total rewards, and timesteps, the suggested method is thoroughly tested in the TORCS Racing Simulator. The findings show that combining FPGA-based hardware acceleration with DDQN successfully improves computational efficiency and decision-making reliability, tackling significant issues brought on by uncertainty in autonomous driving CPSs. In addition to advancing reinforcement learning applications in CPSs, this work opens up possibilities for future investigations into real-world generalisation, adaptive reward mechanisms, and scalable hardware implementations to further reduce uncertainty in autonomous systems.

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

In order to better understand state-of-the-art reinforcement learning agent, deep Q-network, a brief review of reinforcement learning and Q-learning are first described. Then recent advances of deep Q-network are presented, and double deep Q-network and dueling deep Q-network that go beyond deep Q-network are also given.

<div style="text-align: start;" align="center"><span lang="EN-US">The fast emerging of internet of things (IoTs) has introduced fog computing as an intermediate layer between end-users and the cloud datacenters. Fog computing layer characterized by its closeness to end users for service provisioning than the cloud. However, security challenges are still a big concern in fog and cloud computing paradigms as well. In fog computing, one of the most destructive attacks is man-in-the-middle (MitM). Moreover, MitM attacks are hard to be detected since they performed passively on the network level. This paper proposes a MitM mitigation scheme in fog computing architecture. The proposal mapped the fog layer on software-defined network (SDN) architecture. The proposal integrated multi-path transmission control protocol (MPTCP), moving target defense (MTD) technique, and reinforcement learning agent (RL) in one framework that contributed significantly to improving the fog layer resources utilization and security. The proposed schema hardens the network reconnaissance and discovery, thus improved the network security against MitM attack. The evaluation framework was tested using a simulation environment on mininet, with the utilization of MPTCP kernel and Ryu SDN controller. The experimental results shows that the proposed schema maintained the network resiliency, improves resource utilization without adding significant overheads compared to the traditional transmission control protocol (TCP).</span></div>

A deep reinforcement learning based algorithm for a distributed precast concrete production scheduling