Timing Strategy for Active Detection of APT Attack Based on FlipIt Model and Q-learning Method

At present, APT attack detection has become the focus of the network security protection field. APT attacks are one of the most difficult attacks in cyber attacks. The complexity and variability of APT attack behavior greatly increases the difficulty of attack detection. In order to cope with APT attack, some well-known network security companies at home and abroad have developed a commercial APT intrusion detection system. This highly targeted attack can not be identified by the traditional intrusion detection system. Therefore, in order to deal with this new type of cyber attack. The paper proposes a new method to detect APT attack from different organizations. Data mining algorithm is used to analyze every organization’s APT network attack behavior and obtain association rules, so as to customize the design of the Snort rules and apply them to intrusion detection system. Experiments have shown that the evaluation index of the intrusion detection system using the extended Snort rule is significantly better than the traditional Snort intrusion detection system when detecting the same test data. The precision of the extended Snort intrusion detection system is as high as 98.3%, and the false alarm rate is almost 0, which ultimately achieves the purpose of APT detection.

Cyberspace has been threatened by attacks ever since its birth. With the development of the information technologies, especially big data and artificial intelligence, many kinds of cyber attacks are emerging every day, causing severe consequences to society. Meanwhile, intelligent defense methods are proposed to detect these attacks. Such attack and defense methods are constantly being renovated. In particular, advanced persistent threats are intensifying. How to effectively prevent this type of attack has become the a vital problem in recent years. The detection and defense technologies have made great progress. This chapter mainly discusses the research progress of APT attack detection and defense strategies at home and abroad, and focuses on the practice of using machine learning methods to perform attack detection while elaborating on traditional attack detection methods. We also introduce game theory based defense strategy to find the best defense strategy in limited resources, dynamic information flow tracking and cloud platform. With the development of knowledge representation, how to use the MDATA model to characterize the APT attacks is also be discussed in this chapter.KeywordsMDATACyber securityAPT attackCyber defense

The APT attack on the Internet is becoming more serious, and most of intrusion detection systems can only generate alarms to some steps of APT attack and cannot identify the pattern of the APT attack. To detect APT attack, many researchers established attack models and then correlated IDS logs with the attack models. However, the accuracy of detection deeply relied on the integrity of models. In this paper, we propose a new method to construct APT attack scenarios by mining IDS security logs. These APT attack scenarios can be further used for the APT detection. First, we classify all the attack events by purpose of phase of the intrusion kill chain. Then we add the attack event dimension to fuzzy clustering, correlate IDS alarm logs with fuzzy clustering, and generate the attack sequence set. Next, we delete the bug attack sequences to clean the set. Finally, we use the nonaftereffect property of probability transfer matrix to construct attack scenarios by mining the attack sequence set. Experiments show that the proposed method can construct the APT attack scenarios by mining IDS alarm logs, and the constructed scenarios match the actual situation so that they can be used for APT attack detection.

Path planning is an essential algorithm in any autonomous mobile robot, including agricultural robots. One of the reinforcement learning methods that can be used for mobile robot path planning is the Q-Learning algorithm. However, the conventional Q-learning method explores all possible robot states in order to find the most optimum path. Thus, this method requires extensive computational cost especially when there are considerable grids to be computed. This study modified the original Q-Learning algorithm by removing the impassable area, so that these areas are not considered as grids to be computed. This modified Q-Learning method was simulated as path finding algorithm for autonomous mobile robot operated at the Agribusiness and Technology Park (ATP), IPB University. Two simulations were conducted to compare the original Q-Learning method and the modified Q-Learning method. The simulation results showed that the state reductions in the modified Q-Learning method can lower the computation cost to 50.71% from the computation cost of the original Q-Learning method, that is, an average computation time of 25.74s as compared to 50.75s, respectively. Both methods produce similar number of states as the robot’s optimal path, i.e. 56 states, based on the reward obtained by the robot while selecting the path. However, the modified Q-Learning algorithm is capable of finding the path to the destination point with a minimum learning rate parameter value of 0.2 when the discount factor value is 0.9.

Currently many non-tractable considered problems have been solved satisfactorily through methods of approximate optimization called metaheuristic. These methods use non-deterministic approaches that find good solutions which, however, do not guarantee the determination of the global optimum. The success of a metaheuristic is conditioned by capacity to adequately alternate between exploration and exploitation of the solution space. A way to guide such algorithms while searching for better solutions is supplying them with more knowledge of the solution space (environment of the problem). This can to be made in terms of a mapping of such environment in states and actions using Reinforcement Learning. This paper proposes the use of a technique of Reinforcement Learning - Q-Learning Algorithm - for the constructive phase of GRASP and Reactive GRASP metaheuristic. The proposed methods will be applied to the symmetrical traveling salesman problem.

Path planning is a fundamental method in solving mazes or moving robots traversing through open fields with obstacles. Q-learning method is a model-independent reinforcement learning method, which can be utilized in path planning optimization for robots in a multi-robot collaboration system. However, the efficiency of the traditional Q-learning algorithm is relatively low because of the adopted random exploration policy. In this paper, a Boltzmann Policy based Q-learning algorithm is proposed and applied into the problem of path planning optimization of a Multi-robot system. The method is composed of two parts, which are Q-learning and Boltzmann policy. Q-learning is a grid-based algorithm that can solve the low-dimensional path planning problems. Boltzmann Policy adopts statistical probability and simulated annealing, so it can help to avoid trapping in local optimum and provide global optimal solution. Player/Stage is used to evaluate the performance, which shows that the proposed Q-learning algorithm based on the Boltzmann policy can remarkably improve the efficiency of the multi-robot System, reducing the number of explorations and converging the process.

Investors who intend to execute large orders have to always a trade-off between price impact and opportunity cost in Big Data. In this study, reinforcement learning (Q-learning) is applied, due to its strength to support an agent, to make the best decision and take suitable action in a dynamic environment to achieve an optimal way to execute a large number of orders in a day trade. Through Q-learning algorithm, the agent has learned the kind of order yet by how much of it should be submitted in each step to achieve the optimum volume-weighted average price, as the objective function of learning in this study. Historical data of shares in Tehran Stock Exchange has been used to consider the possibility of order types to set the parameters of the simulated trading market, and the price impact on large orders has also been considered through this simulated Big Data to make it with higher accuracy. Results show that for a large order both on buy-side and sell-side separately, execution strategy adopting with multiple order types could be more appropriate compared with using a single order type of execution strategy. This case study suggests that passive order type leads the trader to achieve better results. Compared to the market, the optimal strategy has managed to reduce the volume weighted average price (transaction costs) by 0.95 percent on buy-side and increase it 1.31 percent on sell-side.

Contemporary behavioral learning theory provides a comprehensive description of how we and other animals learn, and places behavioral flexibility and automaticity at heart of adaptive behaviors. However, to our knowledge, the computations supporting the interactions between deliberative and habitual decision-making systems are still poorly understood. Previous functional magnetic resonance imaging (fMRI) results suggest that the dorsal striatum host complementary computations that may differentially support deliberative and habitual processes [1] in the form of a dynamical interplay rather than a serial recruitment of strategies. From the same instrumental task, we develop a dual-system computational model of the two systems that can predict both performance (i.e., participant choices) and modulations in reaction times during learning. The instrumental task is a trial-and-error learning task requiring participants to find the correct associations between color stimuli and finger responses. To model the habitual system, we use a simple Q-learning algorithm (QL) [2] whose properties are fast responses, but slow convergence. For the deliberative (i.e goal-directed) system, we propose a new Bayesian Working Memory (BWM) which searches for information in the history of previous trials and stops as soon as the uncertainty on the action to perform decreases below a certain threshold. Last, we also propose a model for QL and BWM coordination. Currently, most models of system selection tend to control action selection concurrently, using either the deliberative or habitual model according to uncertainty criteria [3,4]. Only one model has investigated the relation between working memory and reinforcement learning [5] without, however explicitly modeling the temporal aspect of memory manipulation. In our approach, we propose a model for QL and BWM coordination. QL and BWM are merged such that the expensive memory manipulation is under control of, among others, the level of convergence of the habitual learning. Consequently, we also predict specific reaction times for each model that can be compared with the evolution of reaction times in instrumental learning tasks. Models are optimized for each subject with the NSGA-2 multi-objective evolutionary algorithm. The first fitness function is the Bayesian Information Criterion for individual choices. The second fitness function is also a likelihood that maximizes the probability of performing reaction times similar to humans. We compare the ability of the new model to explain human behavior with the QL or BWM only, as well as with a combination of these models based on [4], which reveals that the proposed model is in general more accurate. To conclude, we suggest that a close combination of BWM and QL better explains both choices and reaction times for most participants.

In the cybersecurity attack and defense space, the “attacker” and the “defender” form a dynamic and symmetrical adversarial pair. Their strategy iterations and capability evolutions have long been in a symmetrical game of mutual restraint. We will introduce modern Intrusion Detection Systems (IDSs) from the defender’s side to counter the techniques designed by the attacker (APT attack). One major challenge faced by IDS is to identify complex attack paths from a vast provenance graph. By constructing an attack behavior tracking graph, the interactions between system entities can be recorded, but the malicious activities of attackers are often hidden among a large number of normal system operations. Although traditional methods can identify attack behaviors, they only focus on the surface association relationships between entities and ignore the deep causal relationships, which limits the accuracy and interpretability of detection. Existing graph anomaly detection methods usually assign the same weight to all interactions, while we propose a Causal Autoencoder for Graph Explanation (CAGE) based on reinforcement learning. This method extracts feature representations from the traceability graph through a graph attention network(GAT), uses Q-learning to dynamically evaluate the causal importance of edges, and highlights key causal paths through a weight layering strategy. In the DARPA TC project, the experimental results conducted on the selected three datasets indicate that the precision of this method in the anomaly detection task remains above 97% on average, demonstrating excellent accuracy. Moreover, the recall values all exceed 99.5%, which fully proves its extremely low rate of missed detections.

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

Reinforcement learning is a framework in which an agent can learn behavior without knowledge on a task or an environment by exploration and exploitation. Striking a balance between exploration and exploitation is one of the key problems of action selection in reinforcement learning. Exploitation causes the agent to reach a locally optimal policy quickly, whereas excessive exploration degrades the performance of the algorithm, though it may improve the learning performance and escape from a locally optimal policy. Recently the human immune systems have aroused researcher's interest due to its useful mechanisms which can be exploited for information processing in a complex cognition system. In this paper, we transplant some immune mechanisms into the basic Q-learning algorithm and convert Q-learning algorithm into a search for the optimum solution in combinatorial optimization. Experiments show that the improved Q-learning converges more quickly than Q-learning or Boltzmann exploration, and easily obtains the global solution set.

The problem of path exploration and planning of agents in unknown environments is a popular application problem in the field of reinforcement learning. In this paper, we propose an improved reinforcement learning algorithm called the Q-Learning algorithm for adaptive exploration based on simulated annealing (AE-SAQL). We apply the algorithm to the agent path planning problem, improve the setting of the reward function and add the feedback information of the environment. By simulating the Metropolis criterion in the annealing algorithm and adding an adaptive adjustment mechanism, the agent fully explores the environment and makes full use of the environmental information, solving the exploration-utilization dilemma during the algorithm and finally enabling the agent to reach the target location safely. Compared with the standard Q-Learning algorithm and SARSA algorithm, AE-SAQL achieves better.

Reinforcement learning approach for resource allocation in humanitarian logistics

Mobile ad hoc networks (MANETs) consist of a set of nodes which can move freely and communicate with each other wirelessly. Due to the movement of nodes and unlike wired networks, the available routes used among the nodes for transmitting data packets are not stable. Hence, proposing real-time routing protocols for MANETs is regarded as one of the major challenges in this research domain. Algorithms compatible with the changes created in the network due to the nodes' movements are of high significance. For reducing data packet transmission time among nodes, not only should route shortness be considered but also route stability should be taken into consideration. Since available factors in different environments have specific behavior patterns especially in human environments, the parameters of link stability and route shortness were taken into consideration and the reinforcement learning was used to propose a method so as to make the best choice among the neighbors at any moment to transmit a packet to the destination. That is, the proposed method was aimed at predicting the behavior pattern of the nodes in relation to the target node through using reinforcement learning. The proposed method used Q-learning algorithm which has more homogeneity to estimate the value of actions. Simulation results in OPNET demonstrate the superiority of the proposed scheme over conventional MANET routing methods.