Intrusion detection in big data using hybrid feature fusion and optimization enabled deep learning based on spark architecture

Chapter 5 - Intrusion Prevention and Detection Systems

In this paper, by combining the known deep learning related literature in the field of intrusion detection, and the existing in the design of intrusion detection system based on the deep learning common low detection efficiency and high rate of false positives, as well as the training data set used by the problems of unbalanced data, etc., through the depth study of feature extraction and data processing ability. To study the intrusion detection method of the Internet of Things based on deep learning. The work and achievements are as follows:First of all, will be used for feature selection embedded model (EM) and used in intrusion detection of the convolution neural network (CNN), combined with lightweight intrusion detection model (XCNN), because it is an embedded model (EM), so can reduce the impact on the equipment, at the same time compared with other algorithms, the computing time is shorter, the same or even better.Second, proposed to the Attention mechanism (Self - Attention) is applied to intrusion detection system, due to the nature of the Attention mechanism, the mechanism and long Attention can greatly reduce the training time, through introducing the residual network, application feature coding technique again the sparse features thickened into low density at high altitudes, and join to its relative position encoding, So it can sense global information and solve the long-term dependence problem of recurrent neural network.Finally, the results of the intrusion detection method were analyzed on the Pthon simulation platform, and the relevant indicators were verified by using THE NSL-KDD dataset, THE CIC-IDS2017 dataset and the CSE-CIC-IDS2018 dataset. The effectiveness of the method was proved by building a laboratory Internet of Things environment for actual testing. The experimental results show that the algorithm can greatly reduce training time, improve training efficiency, and have the same or even better performance results.

With the continuous development of computer networks, the security of the network has become increasingly prominent. A major threat to network security is the intrusion of information systems through the network. Intrusion detection of the traditional intrusion detection and alarm technology is not sufficient. Based on neural network technology, this paper studies the intrusion detection and alarm correlation technology. Based on the research on the working principle and workflow of the existing intrusion detection system, a new neural network-based intrusion detection and alarm method is proposed. A neural network-based intrusion detection and alarm system is designed and implemented. Through the experiment of the system prototype, the results show that the intrusion detection and alarm system based on the neural network has a higher detection rate and a lower false alarm rate for intrusion behaviors such as denial of service attack and has higher detection ability for unknown attack behaviors.

This paper proposes an effective classification method named Rider Chicken Optimization Algorithm-based Recurrent Neural Network (RCOA-based RNN) to perform big data classification in spark architecture. Initially, the input data are collected from the network by the master node and then forwarded to the slave node. These nodes are responsible for storing the data and performing computations. The features are effectively selected in the slave node using the proposed RCOA. The selected features are forwarded to the master node. The big data classification is achieved in the master node by using the RNN classifier, and the training of the classifier is done using the proposed RCOA algorithm, which is the integration of the Rider optimization algorithm (ROA) with the standard Chicken Swarm Optimization (CSO). The experimentation is done by using the Switzerland dataset, Cleveland dataset, Hungarian dataset and Skin disease dataset, in which the proposed RCOA-based RNN attained better performance based on the quantitative properties, such as sensitivity, accuracy and specificity with the values of 9.3E+01%, 9.4E+01% and 9.3E+01% using Hungarian dataset. The existing learning methods failed to address the complex classification problems at a reasonable time, which is overcome by the proposed method.

Blockchain and federated learning-based intrusion detection approaches for edge-enabled industrial IoT networks: a survey

Intrusions in an information system are the activities that violate the security policy of the system, and intrusion detection is the process used to identify intrusions. Intrusion detection has been studied for approximately 20 years. It is based on the beliefs that an intruder’s behavior will be noticeably different from that of a legitimate user and that many unauthorized actions will be detectable. Intrusion detection systems (IDSs) are usually deployed along with other preventive security mechanisms, such as access control and authentication, as a second line of defense that protects information systems. There are several reasons that make intrusion detection a necessary part of the entire defense system. This paper describes various Intrusion Detection methods like pattern matching, state full pattern matching, protocol decode-based analysis etc and how fuzzy clustering can apply in IDS.

Cloud computing (CC) is an internet-enabled environment that provides computing services such as networking, databases, and servers to clients and organizations in a cost-effective manner. Despite the benefits rendered by CC, its security remains a prominent concern to overcome. An intrusion detection system (IDS) is generally used to detect both normal and anomalous behavior in networks. The design of IDS using a machine learning (ML) technique comprises a series of methods that can learn patterns from data and forecast the outcomes consequently. In this background, the current study designs a novel multi-objective seagull optimization algorithm with a deep learning-enabled vulnerability detection (MOSOA-DLVD) technique to secure the cloud platform. The MOSOA-DLVD technique uses the feature selection (FS) method and hyperparameter tuning strategy to identify the presence of vulnerabilities or attacks in the cloud infrastructure. Primarily, the FS method is implemented using the MOSOA technique. Furthermore, the MOSOA-DLVD technique uses a deep belief network (DBN) method for intrusion detection and its classification. In order to improve the detection outcomes of the DBN algorithm, the sooty tern optimization algorithm (STOA) is applied for the hyperparameter tuning process. The performance of the proposed MOSOA-DLVD system was validated with extensive simulations upon a benchmark IDS dataset. The improved intrusion detection results of the MOSOA-DLVD approach with a maximum accuracy of 99.34% establish the proficiency of the model compared with recent methods.

Cloud computing is the distributed computing paradigm continually exposed to different attacks and threats of various origins. The data stored in the cloud framework is easier for external and internal intruders, as access to the cloud framework is done through internet services. Various intrusion detection (ID) methods are developed to detect network intruders in the cloud, but these methods are not primarily effective in generating accurate detection results. Hence, an effective intrusion detection system (IDS) is designed to solve the security issues that unfavorably influence the sustainable development of the cloud and enhance the protection of the cloud from malicious attacks. The IDS is modeled using the proposed Feedback Deer Hunting Optimization (FDHO)-based Deep Residual network to detect network intrusions. However, the proposed FDHO algorithm is designed by integrating Feedback Artificial Tree (FAT) with Deer Hunting Optimization (DHOA), respectively. Moreover, the detection of malicious attacks is carried out using a Deep Residual network that significantly increases the training speed, reduces the computational complexity, and generates effective detection results. The performance of the proposed method is comparatively analyzed with the existing techniques, such as Stacked Contractive Auto-Encoder and Support Vector Machine (SCAE+SVM), Artificial Neural Network with ant bee colony optimization algorithm+fuzzy clustering (ANN+ABC+fuzzy clustering), Improved dynamic immune algorithm (IDIA), and Normalized K-means (NK) clustering algorithm with RNN named, (NK-RNN), FAT-based Deep Residual network, and DHOA-based Deep Residual network using the BoT-IoT dataset and KDD cup-99 dataset. The proposed method achieved outstanding performance by considering the metrics, like specificity, accuracy, and sensitivity, with the values of 0.9526, 0.9498, and 0.9214 using the BoT-IoT dataset.

With the popularization and development of network knowledge, network intruders are increasing, and the attack mode has been updated. Intrusion detection technology is a kind of active defense technology, which can extract the key information from the network system, and quickly judge and protect the internal or external network intrusion. Intrusion detection is a kind of active security technology, which provides real-time protection for internal attacks, external attacks and misuse, and it plays an important role in ensuring network security. However, with the diversification of intrusion technology, the traditional intrusion detection system cannot meet the requirements of the current network security. Therefore, the implementation of intrusion detection needs diversifying. In this context, we apply neural network technology to the network intrusion detection system to solve the problem. In this paper, on the basis of intrusion detection method, we analyze the development history and the present situation of intrusion detection technology, and summarize the intrusion detection system overview and architecture. The neural network intrusion detection is divided into data acquisition, data analysis, pretreatment, intrusion behavior detection and testing.

The article gives a general idea of intrusion detection systems (IDS) and machine learning, presents a classification of machine learning methods. The most commonly used data sets for training IDS are considered. The main performance indicators used to assess the IDS are described. A review of modern research on the development of intrusion detection methods using machine learning is made, the strengths and weaknesses of each solution are described. The data sets used in them, the obtained estimates of the effectiveness of the proposed intrusion detection methods, are analyzed. Based on the results of the analysis, development trends, unresolved problems and promising areas of research were identified.

The novelty and growing sophistication of cyber threats mean that high accuracy and interpretable machine learning models are needed more than ever before for Intrusion Detection and Prevention Systems. This study aims to solve this challenge by applying Explainable AI techniques, including Shapley Additive explanations feature selection, to improve model performance, robustness, and transparency. The method systematically employs different classifiers and proposes a new hybrid method called Hybrid Bagging-Boosting and Boosting on Residuals. Then, performance is taken in four steps: the multistep evaluation of hybrid ensemble learning methods for binary classification and fine-tuning of performance; feature selection using Shapley Additive explanations values retraining the hybrid model for better performance and reducing overfitting; the generalization of the proposed model for multiclass classification; and the evaluation using standard information metrics such as accuracy, precision, recall, and F1-score. Key results indicate that the proposed methods outperform state-of-the-art algorithms, achieving a peak accuracy of 98.47% and an F1 score of 96.19%. These improvements stem from advanced feature selection and resampling techniques, enhancing model accuracy and balancing precision and recall. Integrating Shapley Additive explanations-based feature selection with hybrid ensemble methods significantly boosts the predictive and explanatory power of Intrusion Detection and Prevention Systems, addressing common pitfalls in traditional cybersecurity models. This study paves the way for further research on statistical innovations to enhance Intrusion Detection and Prevention Systems performance.

Survey of federated learning in intrusion detection

As an active network security protection scheme, intrusion detection system (IDS) undertakes the important responsibility of detecting network attacks in the form of malicious network traffic. Intrusion detection technology is an important part of IDS. At present, many scholars have carried out extensive research on intrusion detection technology. However, developing an efficient intrusion detection method for massive network traffic data is still difficult. Since Generative Adversarial Networks (GANs) have powerful modeling capabilities for complex high-dimensional data, they provide new ideas for addressing this problem. In this paper, we put forward an EBGAN-based intrusion detection method, IDS-EBGAN, that classifies network records as normal traffic or malicious traffic. The generator in IDS-EBGAN is responsible for converting the original malicious network traffic in the training set into adversarial malicious examples. This is because we want to use adversarial learning to improve the ability of discriminator to detect malicious traffic. At the same time, the discriminator adopts Autoencoder model. During testing, IDS-EBGAN uses reconstruction error of discriminator to classify traffic records.

As the development of intelligent ships rapidly progresses, the importance of maritime network security escalates. Intrusion Detection Systems (IDS) serve as a pivotal defense mechanism against cyberattacks targeting these vessels. To augment the efficacy of IDS in detecting anomalous network traffic, this paper introduces a novel intrusion detection method utilizing both Transformer and Kolmogorov-Arnold Networks (KAN), designated as Transformer-KAN. This approach effectively mitigates the limitations of traditional intrusion detection algorithms, which include inadequate feature extraction capabilities and the excessive parameterization required to delineate complex patterns. The proposed method employs a Transformer encoder to discern long-range dependencies within input sequences, while KAN layers enhance the model’s ability to approximate complex patterns through non-linear transformations. This integrative strategy ensures elevated accuracy for the intrusion detection algorithm and proves particularly beneficial in intelligent ship environments, where training data may be sparse or computational resources limited. Experimental results confirm that the Transformer-KAN model attains exceptionally high accuracy in both binary and five-class classification tasks, outperforming the capabilities of standalone Transformer and KAN models. Comparative analyses with conventional algorithms—such as Convolutional Neural Networks (CNN), Gated Recurrent Units (GRU), Deep Belief Networks (DBN), and Support Vector Machines (SVM)-further substantiate the superior effectiveness of the proposed method. This research establishes a new theoretical framework and provides a direction for the practical implementation of network security measures in contemporary intelligent ships.

With the industrial revolution 4.0, the use of IoT-based systems is increasing, both in the field of health manufacturing, urban planning, housing, and even automotive. Therefore, the security of the IoT system needs to be considered, this is related to data integrity, privacy, service stability. Through intrusion detection, activities on the IoT system will be able to be analyzed whether there are suspicious activities that can interfere with or threaten IoT services. In several previous studies in the literature, the approach used to detect intrusions in the IoT system has a high false alarm rate. This research proposes an approach through machine learning, specifically the ensemble learning approach and the synthetic minority over-sampling technique (SMOTE) method as a method of detecting intrusions in the IoT system which is expected to produce better performance. The results of this study indicate that the proposed approach is able to detect intrusion and classify into five types of intrusion including normal intrusion, probe, dos, r2l, u2r. Based on the evaluation results, the proposed approach can improve the performance of intrusion detection in terms of accuracy to 97.02%, detection rate of 97%, false alarm rate 0.16%, compared to base learning and approaches in previous studies used as intrusion detection methods, but in processing time performance have not shown satisfying results.