Deep Reinforcement Learning-Based Traffic Sampling for Multiple Traffic Analyzers on Software-Defined Networks

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

The increasing demand for fast, reliable, and robust network services has driven the telecommunications industry to design novel network architectures that employ Network Functions Virtualization and Software Defined Networking. Despite the advancements in cellular networks, there is a need for an automatic, self-adapting orchestrating mechanism that can manage the placement of resources. Deep Reinforcement Learning can perform such tasks dynamically, without any prior knowledge. In this work, we leverage a Deep Deterministic Policy Gradient Reinforcement Learning algorithm, to fully automate the Virtual Network Functions deployment process between edge and cloud network nodes. We evaluate the performance of our implementation and compare it with alternative solutions to prove its superiority while demonstrating results that pave the way for Experiential Network Intelligence and fully automated, Zero touch network Service Management.

A flexible SDN-based framework for slow-rate DDoS attack mitigation by using deep reinforcement learning

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

Attack-aware Security Function Management

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

With the surge in cyber attacks, there is a pressing need for more robust network intrusion detection systems (IDSs). These IDSs perform at their best when they can monitor all the traffic coursing through the network, especially within a software-defined network (SDN). In an SDN configuration, the control plane and data plane operate independently, facilitating dynamic control over network flows. Typically, an IDS application resides in the control plane, or a centrally located network IDS transmits security reports to the controller. However, the controller, equipped with various control applications, may encounter challenges when analyzing substantial data, especially in the face of high traffic volumes. To enhance the processing power, detection rates, and alleviate the controller’s burden, deploying multiple instances of IDS across the data plane is recommended. While deploying IDS on individual switches within the data plane undoubtedly enhances detection rates, the associated costs of installing one at each switch raise concerns. To address this challenge, this paper proposes the deployment of IDS chains across the data plane to boost detection rates while preventing controller overload. The controller directs incoming traffic through alternative paths, incorporating IDS chains; however, potential delays from retransmitting traffic through an IDS chain could extend the journey to the destination. To address these delays and optimize flow distribution, our study proposes a method to balance flow assignments to specific IDS chains with minimal delay. Our approach is validated through comprehensive testing and evaluation using a test bed and trace-based simulation, demonstrating its effectiveness in reducing delays and hop counts across various traffic scenarios.

AQROM: A quality of service aware routing optimization mechanism based on asynchronous advantage actor-critic in software-defined networks

Traffic engineering (TE) can balance the traffic in the network to reduce network congestion and improve network resource utilization. The emergence of Software Defined Network (SDN) provides a more flexible and effective way to control traffic in the network. Existing TE solutions mainly focus on routing traffic via the shortest path or evenly distributing the traffic among multiple available paths, but these methods are not flexible since this static mapping of traffic to paths does not consider either the current network utilization or traffic load. Heuristics-based TE methods depends on operators' understanding of the workload and environment. Designing and implementing those methods thus take at least weeks. Furthermore, it usually takes minutes to output the solution. Inspired by recent successes in applying Deep Reinforcement Learning (DRL) techniques to solve complex control problems, we leverage DRL to control traffic in SDN. We start by building a framework which integrates the DRL algorithm into SDN. Based on this framework, we propose a modified DRL algorithm to control the traffic split ratio to multiple paths. Simulation results show that the proposed approach performs better than three baseline methods when the traffic load is dynamically changing.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

Software defined networks, as an emerging network architecture, separate the control plane and data plane of the network, providing new ideas and methods for network management and optimization.In software defined networks, routing optimization is one of the key issues in improving network performance and flexibility.The current routing optimization algorithms often fail to fully consider the dynamics and complexity of the network, making it difficult to achieve intelligent management and optimization of the network.In order to accurately and timely perceive the dynamic changes of network traffic and improve network stability, this paper combines deep reinforcement learning to conduct in-depth research on routing optimization algorithms in software defined networks.This article first combines DQN (Deep Q Network) and Optimized Link State Routing (OLSR) protocols in deep reinforcement learning, capturing the relationships and traffic directions between different nodes in the network; Then, the link state information used for routing decisions is exchanged to solve the resource waste problem in Deep Deterministic Policy Gradient (DDPG), in order to achieve network optimization and intelligent management of network resources; Finally, based on real-time network status and service quality requirements, the optimal path is dynamically selected to respond to changes in the network environment and provide stable and reliable network services, thereby optimizing routing.The experimental results show that the average throughput of the DQN-OLSR route is 21.9% and 31.4% higher than that of the DQN and DDPG routes, respectively.The conclusion indicates that DQN-OLSR makes the network more flexible and helps to improve the service quality and management level of the entire network.

Irrigation decision-making using Reinforcement Learning (RL) performs well in changing environment, but easily falls into sub-optimal solutions with high-dimensional data. Deep Reinforcement Learning (DRL) has fused RL with Deep Learning (DL) and excels at learning adaptive and long-term irrigation strategies directly from high-dimensional environment data. This paper systematically reviews the applications of DRL in irrigation optimization, covering both pre-trained environments based on crop growth simulators and dynamic environments driven by real-time sensors. We discussed the strengths of classic DRL algorithms, including their ability to handle dynamic and non-linear environments, and reviewed their performance in irrigation multi-objective optimization and decision-making. In addition, we identified constraints in applying DRL in irrigation decision making, which include data scarcity, poor model interpretability, and difficulties in field deployment. It shows DRL can provide a powerful framework for adaptive irrigation, but is constrained by the gap between simulation and real-world complexity. To address these limitations, we discussed approaches in future work, such as developing multi-objective DRL algorithms. These approaches will improve DRL modeling outcomes and provide a technological foundation for smart agriculture and sustainable resource management. • Review the application of Deep Reinforcement Learning (DRL) in agricultural irrigation. • Analyze the performance of DRL algorithms in irrigation decision-making. • Compare DRL models based on different environment in irrigation optimization. • Discuss the further work to improve DRL performance in irrigation optimization.

Deep Reinforcement Learning (DRL) algorithm provides an effective and powerful way for the computer learning to make decisions according to what the computer sees, the most famous application of DRL is training the computer to play computer games, such as the computer Go and Atari Games. In this paper, we present that the DRL can used to improve the classical PID controller by training an adaptive PID controller with its parameters change according to the changing state, we name our new controller as DRPID, the deep reinforcement learning algorithm that we used to train the PID controller in this paper is deep deterministic policy gradient, which is very suitable to solve continue action control problem. We test our DRPID algorithm by training it to control an inverted pendulum in the OpenAI gym simulation environment, and we found DRPID controller works very well, outperforms the common PID controller with fixed parameters, by a great margin.

The increasing complexity and connectivity of networks lead to new and challenging vulnerabilities to be addressed for network security. As networks expand and diverse networks become interrelated, attackers leverage the broad scope of the global Internet to perpetrate large-scale attacks. Cloud environments, which are widely adopted, are fundamentally provisioned via the Internet bringing new challenges regarding security and privacy. However, network programmability and emerging technologies such as software-defined networking (SDN) and network functions virtualization (NFV) can advance techniques for intrusion detection and protection. In this chapter, we review the state of the art in scalable and collaborative intrusion detection systems (IDSs). Combining data plane programming with control level collaboration, we propose a model to simplify the detection of large-scale, distributed network attacks. This model simultaneously reduces the system overhead through direct mitigation at the network edge and enriches the detection process with corroboration of evidence from distributed sources.