Anomaly detection in Network Traffic Using Unsupervised Machine learning Approach

Systems employing the Internet and Voice over Internet Protocol (VoIP) are alternative to the legacy landline telephony system. The services offered by these systems allow users to communicate with their family members, friends, banks and business partners whenever they are online. Particularly, the VoIP service is popular with the Internet users because charges are bound into usually fixed access costs making the price of the long distance calls themselves appear economical and even free. However, VoIP services have several disadvantages. One of the disadvantages of VoIP is, it is exposed to Internet security vulnerabilities, threats and attacks. The attacks come in many forms. One of the methods to attack the VoIP service, is by sending a Spam over Internet Telephony (SPIT). SPIT is normally executed by malicious parties initiating automated, unsolicited, and unwanted communications that used VoIP or video conferencing services, just like an email SPAM. Solutions to mitigate SPIT are still lacking. To defense a VoIP system from SPIT, we have come out with the idea of Automated Intrusion Detection and Prevention System over SPIT (AIDPoS) that can detect and prevent SPIT from external parties to enter the VoIP network. AIDPoS is a combination system of Intrusion Detection System (IDS) and Intrusion Prevention System (IPS) on Voice over IP (VoIP) server. This is an experimental research to find a solution to combat SPIT if it happens in the future. Our result shows that an automated Intrusion detection and Prevention system over SPIT attack is feasible. Our work is on an open source VoIP network. Other vulnerabilities and threats are important, but beyond our research scope and is not covered in this paper.

Chapter 5 - Intrusion Prevention and Detection Systems

Malware authors are continuously developing crime toolkits. This has led to the situation of zero-day attacks, where malware harm computer systems despite the protection from existing Intrusion Detection Systems (IDSs). We propose an Intelligent Intrusion Detection and Prevention System (IIDPS) approach that combines the Signature based Intrusion Detection system (SIDS), Anomaly based Intrusion Detection System (AIDS) and Response Intrusion Detection System (RIDS). We used a risk assessment approach to determine an appropriate response action against each attack event. We also demonstrated the IIDPS make the detection and prevention of malware more effective.

Nowadays, cyber-attacks are beginning to be smart and hard to be detected, these attacks can be classified as Advance Persistence Threat(APT), which are hard to be detected and need sophisticated mechanisms of cybersecurity detection systems to be implemented to detect these types of attacks. Snort is an Intrusion Detection System (IDS) open source application that gained a high level of trust from hundreds of companies by using it as an IDS sensor whether Host-based Intrusion Detection System (HIDS) or Network Intrusion Detection System (NIDS) mode. Snort is depending on rules that are predefined to detect known attacks where if there is a new attack released and have not been registered to CheckPoint as an attack, then Snort will not detect it as an attack and the attack will bypass. The main problem that stands with traditional IDS (Snort) is the rate of false positive alerts. A new technique has been revealed by understanding the behavior of the traffic flow and deciding if the traffic flow matches the attributes that have abnormal activities that the traditional misuse IDS cannot detect. The new technique is a machine learning process that depending on training and testing the data of the traffic after converting the PCAP file to CSV file through an application that calls CICFlowMeter. it is an open source machine learning application that used as GUI mode calls Weka. After the PCAP file converted through CICFlowMeter, a dataset end with CSV extension will be generated with 80 plus attributes that Weka will learn as the training phase after that the testing phase will determine the matched traffic if they are normal or abnormal.

The significant increase in technology development over the internet makes network security a crucial issue. An intrusion detection system (IDS) shall be introduced to protect the networks from various attacks. Even with the increased amount of works in the IDS research, there is a lack of studies that analyze the available IDS datasets. Therefore, this study presents a comprehensive analysis of the relevance of the features in the KDD99 and UNSW-NB15 datasets. Three methods were employed: a rough-set theory (RST), a back-propagation neural network (BPNN), and a discrete variant of the cuttlefish algorithm (D-CFA). First, the dependency ratio between the features and the classes was calculated, using the RST. Second, each feature in the datasets became an input for the BPNN, to measure their ability for a classification task concerning each class. Third, a feature-selection process was carried out over multiple runs, to indicate the frequency of the selection of each feature. From the result, it indicated that some features in the KDD99 dataset could be used to achieve a classification accuracy above 84%. Moreover, a few features in both datasets were found to give a high contribution to increasing the classification’s performance. These features were present in a combination of features that resulted in high accuracy; the features were also frequently selected during the feature selection process. The findings of this study are anticipated to help the cybersecurity academics in creating a lightweight and accurate IDS model with a smaller number of features for the developing technologies.

The Internet is the darling of the information age. Its rapid development has brought a lot of wealth to the society, but it has also brought about increasingly serious security problems. In particular, the widespread application of e-commerce has made network security the most important part of the development of network technology. Network security technology is increasingly valued by computer researchers. Most of the existing security products are passive. The firewall can only be statically disabled. Although the Intrusion Detection System can dynamically detect the intrusion behavior, it cannot block the detected attack behavior. However, the impact of the intrusion prevention system on network performance and the high intrusion detection false alarm rate restrict the development and application of the IPS system. Especially on gigabit high-speed networks, performance has become the primary bottleneck of IPS systems. In view of the above problems, this paper proposes a distributed intrusion detection and blocking system that can run efficiently on high-speed networks. Such systems are also known as intrusion prevention systems IPS. Explain the structure of distributed high-speed network intrusion prevention system, and also introduce the related implementation technology and the realization of each part of distributed intrusion prevention system.

IntroductionIn swine disease surveillance, obtaining labeled data for supervised learning models can be challenging because many farms lack standardized diagnostic routines and consistent health monitoring systems. Unsupervised learning is particularly suitable in such scenarios because it does not require labeled data, allowing for detecting anomalies without predefined labels. This study evaluates the effectiveness of unsupervised machine learning models in detecting anomalies in productivity indicators in swine breeding herds.MethodsAnomalies, defined as deviations from expected patterns, were identified in indicators such as abortions per 1000 sows, prenatal losses, preweaning mortality, total born, liveborn, culled sows per 1000 sows, and dead sows per 1000 sows. Three unsupervised models - Isolation Forest, Autoencoder, and K-Nearest Neighbors (KNN) - were applied to data from two swine production systems. The herd-week was used as the unit of analysis, and anomaly scores above the 75th percentile were used to flag anomalous weeks. A permutation test assessed differences between anomalous and non-anomalous weeks. Performance was evaluated using F1-score, precision, and recall, with true anomalous weeks defined as those coinciding with reported health challenges, including porcine reproductive and respiratory syndrome (PRRS) and Seneca Valley virus outbreaks. A total of 8,044 weeks were analyzed.ResultsThe models identified 336 anomalous weeks and 1,008 non-anomalous weeks in Production System 1, and 1,675 anomalous weeks and 5,025 non-anomalous weeks in Production System 2. The results from the permutation test revealed significant differences in productivity indicators between anomalous and non-anomalous weeks, especially during PRRS outbreaks, with more subtle changes observed during Seneca Valley virus outbreaks. The models performed well in detecting the PRRSV anomaly, achieving perfect precision (100%) across all models for both production systems. For anomalies like SVV the models showed lower performance compared to PRRSV.DiscussionThese findings suggest that unsupervised machine learning models are promising tools for early disease detection in swine herds, as they can identify anomalies in productivity data that may signal health challenges.

Industrial Information Technology infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This article aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detection in industrial big data scenarios. Since our objective is to build the intrusion detection system for the big data scenario in the industrial domain, we utilize the Apache Spark framework to implement our proposed model that was trained in large network traffic data (about 123 million instances of network traffic) stored in Elasticsearch. Moreover, we evaluate our proposed model on the live streaming data and find that our proposed system can be used for real-time anomaly detection in the industrial setup. In addition, we address different challenges that we face while training our model on large datasets and explicitly describe how these issues were resolved. Based on our empirical evaluation in different use cases for anomaly detection in real-world network traffic data, we observe that our proposed system is effective to detect anomalies in big data scenarios. Finally, we evaluate our proposed model on several academic datasets to compare with other models and find that it provides comparable performance with other state-of-the-art approaches.

An intrusion prevention system (IPS) acts as a new type of information security technology, the configuration and management of which are currently urgent problems; in particular, debate exists regarding the value of these systems. In this paper, we analyse whether a firm realizes a positive or negative value from using an IPS instead of an intrusion detection system (IDS) in a default configuration and an optimal configuration, respectively. Our results suggest: (a) an IPS could hurt the firm when not configured optimally; (b) the optimal configuration of the IPS depends not only on the cost parameters but also on the external environment (quality of the IDS) in which the firm is operating; (c) whether the IDS is optimally configured or not, the firm will make the same decisions between using the IPS instead of the IDS and continuing to use the IDS; and (d) except for the true positive rate of IDS being in a certain region and the blocking cost being sufficiently high, the firm realizes a strictly nonnegative value if the firm configures the IPS optimally.

The philosophy of the IoT world is becoming important for a projected, always-connected world. The 5G networks will significantly improve the value of 4G networks in the day-to-day world, making them fundamental to the next-generation IoT device networks. This article presents the current advances in the improvement of the standards, which simulate 5G networks. This article evaluates the experience that the authors gained when implementing Vodafone Romania 5G network services, illustrates the experience gained in context by analyzing relevant peer-to-peer work and used technologies, and outlines the relevant research areas and challenges that are likely to affect the design and implementation of large 5G data networks. This paper presents a machine learning-based real-time intrusion detection system with the corresponding intrusion prevention system. The convolutional neural network (CNN) is used to train the model. The system was evaluated in the context of the 5G data network. The smart intrusion detection system (IDS) takes the creation of software-defined networks into account. It uses models based on artificial intelligence. The system is capable to reveal not previously detected intrusions using software components based on machine learning, using the convolutional neural network. The intrusion prevention system (IPS) blocks the malicious traffic. This system was evaluated, and the results confirmed that it provides higher efficiencies compared to less overhead-like approaches, allowing for real-time deployment in 5G networks. The offered system can be used for symmetric and asymmetric communication scenarios.

Attack-aware Security Function Management

This paper presents Intrusion Detection Systems (IDS), Intrusion Detection and Prevention Systems (IDPS) and their classification emphasizing on the use of neural networks in IDS. Contemporary IDS usually include both signature verification and anomaly detection approaches realized by rule-based expert system and statistical module correspondingly. Neural networks may be used mainly as additional module to the statistical module to better recognize the user behavior. User behavior may be represented as frequency pattern of users command history. The paper presents an example of user profile vector for Unix-based platforms.

The rise of sophisticated cyber threats has spurred advancements in Intrusion Detection Systems (IDS), which are crucial for identifying and mitigating security breaches in real-time. Traditional IDS often rely on complex machine learning algorithms that lack transparency despite their high accuracy, creating a "black box" effect that can hinder the analysts' understanding of their decision-making processes. Explainable Artificial Intelligence (XAI) offers a promising solution by providing interpretability and transparency, enabling security professionals to understand better, trust, and optimize IDS models. This paper presents a systematic review of the integration of XAI in IDS, focusing on enhancing transparency and interpretability in cybersecurity. Through a comprehensive analysis of recent studies, this review identifies commonly used XAI techniques, evaluates their effectiveness within IDS frameworks, and examines their benefits and limitations. Findings indicate that rule-based and tree-based XAI models are preferred for their interpretability, though trade-offs with detection accuracy remain challenging. Furthermore, the review highlights critical gaps in standardization and scalability, emphasizing the need for hybrid models and real-time explainability. The paper concludes with recommendations for future research directions, suggesting improvements in XAI techniques tailored for IDS, standardized evaluation metrics, and ethical frameworks prioritizing security and transparency. This review aims to inform researchers and practitioners about current trends and future opportunities in leveraging XAI to enhance IDS effectiveness, fostering a more transparent and resilient cybersecurity landscape.

The substantial growth of Internet-connected devices within the Internet of Things (IoT) has given rise to significant security challenges. Among the various threats confronting these interconnected devices, Denial of Service (DoS)/Distributed Denial of Service (DDoS) attacks emerge as significant concerns. The attacks, which seek to disrupt IoT services by flooding networks with unnecessary traffic, there is a critical need for robust security measures. Intrusion Detection Systems (IDS) are vital in identifying suspicious activities, yet many existing systems lack real-time capabilities to address evolving attack strategies. This study investigates the vulnerabilities of IoT networks and the pressing need to detect and classify DoS/DDoS attacks in real time. Traditional IDS, while effective in recognizing known attack patterns, fall short in identifying new attack types due to their reliance on historical data. To bridge this gap, this research focuses on developing an enhanced hybrid network intrusion detection and classification system. This study seeks to make contribution to the advancement of resilient security measures within IoT environments. The framework uses the Kth Nearest Neighbor (KNN) algorithm and dense neural networks to efficiently detect and categorize DoS/DDoS attacks in real time. To achieve this, a simulation model implementing the proposed hybrid algorithm will be developed. The framework utilized the MQTT-IoT-IDS2020 dataset, and a comparison was presented with recent approaches found in the literature. The results demonstrated enhanced performance in terms of true positive rate, accuracy, speed of detection.

Intrusion detection is an important part of network security protection. Traditional intrusion detection systems (IDSs) only focus on low-level attacks and raise alerts independently, though there may be logical connections between them. At the same time, the amount of alerts becomes unmanageable including actual alerts mixed with false alerts. Therefore, improved techniques are needed. The general idea is to introduce collaboration achieved by taking advantage of vulnerability analysis as contextual information and thus enable IDSs to correctly identify successful attacks while simultaneously reducing the number of false positives and providing a stronger validation attack scenario. In particular, with the verification pattern with precondition and effect of successful attack and necessary context (mainly modeled as host and connectivity information), the architecture that proposes in this paper can reduce the false alert rate and identify true alerts corresponding to successful attacks to construct attack scenario. Through the experimental results with DARPA Data Sets 2000 from Lincoln laboratory and the Treasure Hunt Dataset, it demonstrates the potential of the proposed techniques.