Big data analysis and distributed deep learning for next-generation intrusion detection system optimization

Taiwan, being located on a path in the west Pacific Ocean where typhoons often strike, is often affected by typhoons. The accompanying strong winds and torrential rains make typhoons particularly damaging in Taiwan. Therefore, we aimed to establish an accurate wind speed prediction model for future typhoons, allowing for better preparation to mitigate a typhoon’s toll on life and property. For more accurate wind speed predictions during a typhoon episode, we used cutting-edge machine learning techniques to construct a wind speed prediction model. To ensure model accuracy, we used, as variable input, simulated values from the Weather Research and Forecasting model of the numerical weather prediction system in addition to adopting deeper neural networks that can deepen neural network structures in the construction of estimation models. Our deeper neural networks comprise multilayer perceptron (MLP), deep recurrent neural networks (DRNNs), and stacked long short-term memory (LSTM). These three model-structure types differ by their memory capacity: MLPs are model networks with no memory capacity, whereas DRNNs and stacked LSTM are model networks with memory capacity. A model structure with memory capacity can analyze time-series data and continue memorizing and learning along the time axis. The study area is northeastern Taiwan. Results showed that MLP, DRNN, and stacked LSTM prediction error rates increased with prediction time (1–6 hours). Comparing the three models revealed that model networks with memory capacity (DRNN and stacked LSTM) were more accurate than those without memory capacity. A further comparison of model networks with memory capacity revealed that stacked LSTM yielded slightly more accurate results than did DRNN. Additionally, we determined that in the construction of the wind speed prediction model, the use of numerically simulated values reduced the error rate approximately by 30%. These results indicate that the inclusion of numerically simulated values in wind speed prediction models enhanced their prediction accuracy.

<p dir="ltr">Recurrent Neural Networks (RNNs) are a major class of Artificial Neural Networks (ANNs). Suitable network architecture is vital for RNNs to achieve high performance. However, designing the architecture of deep RNNs (DRNNs) is a complicated and time-consuming process. Each layer of a DRNN contains numerous hyper-parameters. It is challenging to optimize the hyper-parameters of all layers jointly to achieve the best possible learning performance. Trial-and-error methods for designing DRNNs have been proven to be laborious and highly costly in practice. Therefore, an efficient automatic architecture search technique is needed to design DRNNs.</p><p dir="ltr">Genetic Algorithm (GA) is a popular evolutionary computation based approach. Existing research works already explored the use of GAs in designing ANNs. But, most of the existing approaches focus on the design of fixed-depth ANNs. However, it is not easy to fix the depth of DRNNs in advance, since different problems need DRNNs of different depths. Hence, the potential of GA has to be further explored to evolve the architecture of DRNNs of varying depths.</p><p dir="ltr">The primary goal of this thesis is to develop advanced GA approaches for designing Long Short Term Memory (LSTM) based DRNN architectures of varying depths effectively and efficiently.</p><p dir="ltr">Firstly, this thesis proposes a GA-based algorithm called Two-Stage Surrogate-Assisted GA (TS-SA-GA) with a progressive incremental strategy to design the architecture of LSTM networks of varying depths. The progressive approach can effectively extend well-designed shallow networks to high performing deep networks. Moreover, the new GA-based algorithm adopts newly designed knowledge-driven crossover and mutation operators to identify and repair LSTM network designs affected by inappropriate use of activation functions, thereby significantly reducing the chances for GA to evolve hard-to-train LSTM network architectures. Furthermore, this thesis proposes a two-stage surrogate method to predict the trainability and fitness of LSTM networks. This improves the efficiency and effectiveness of the GA-based algorithm to evolve LSTM networks.</p><p dir="ltr">Secondly, this thesis proposes a new algorithm called Evolutionary Design of LSTM Ensembles (ED-LSTM-Ensemble) based on multi-objective optimization techniques to evolve LSTM networks and ensembles simultaneously to directly utilize the ensemble performance to drive the evolution of base LSTM networks. Additionally, this thesis proposes a connection weight inheritance strategy to evolve LSTM networks and ensembles efficiently and effectively.</p><p dir="ltr">Thirdly, this thesis proposes a new algorithm called Skip Connections through Evolutionary and Differential Architecture search (SCEDA) to design LSTM networks together with appropriate skip connections automatically. Designing LSTM networks together with skip connections is highly challenging. This thesis hybridize gradient-based DAS and evolutionary architecture search to optimize the architecture of LSTM networks with suitable skip connections in a single evolutionary process to improve the efficiency and effectiveness of the search process.</p><p dir="ltr">Finally, this thesis conducts an in-depth empirical analysis of the impact of activation functions on the trainability of LSTM networks based on the concept of Edge Of Chaos. The analysis reveals the strong interrelation between the activation functions across multiple layers of an LSTM network and the trainability of the network. The analysis also highlights the importance of controlling activation functions to avoid generating untrainable LSTM networks. On the basis of the analysis, this thesis proposes a machine learning model to guide the use of activation functions in LSTM networks.</p>

Capturing long-range interactions between structural but not sequence neighbors of proteins is a long-standing challenging problem in bioinformatics. Recently, long short-term memory (LSTM) networks have significantly improved the accuracy of speech and image classification problems by remembering useful past information in long sequential events. Here, we have implemented deep bidirectional LSTM recurrent neural networks in the problem of protein intrinsic disorder prediction. The new method, named SPOT-Disorder, has steadily improved over a similar method using a traditional, window-based neural network (SPINE-D) in all datasets tested without separate training on short and long disordered regions. Independent tests on four other datasets including the datasets from critical assessment of structure prediction (CASP) techniques and >10 000 annotated proteins from MobiDB, confirmed SPOT-Disorder as one of the best methods in disorder prediction. Moreover, initial studies indicate that the method is more accurate in predicting functional sites in disordered regions. These results highlight the usefulness combining LSTM with deep bidirectional recurrent neural networks in capturing non-local, long-range interactions for bioinformatics applications. SPOT-disorder is available as a web server and as a standalone program at: http://sparks-lab.org/server/SPOT-disorder/index.php . j.hanson@griffith.edu.au or yuedong.yang@griffith.edu.au or yaoqi.zhou@griffith.edu.au. Supplementary data is available at Bioinformatics online.

Prediction concept needs human-like thinking and ideas. ANN helps make a program for describing the solutions to the prediction problems. To gain accurate prediction, there is a need for deep thinking in the knowledge system discovered. Deep learning helps form such next-level type of ANN thinking. Deep learning models do not require humans for programming the problems. It can determine the prediction by itself by learning the dataset. This paper describes the prediction by considering sequential model for continuous accessing of data with the help of multi-layers. Deep learning is very helpful for predicting larger dataset. This paper proposes an algorithmic model known as a Comprehensive Deep Recurrent Artificial Neural Networks (CDRANN) by hybridizing LSTM (long short-term memory) with RNN (recurrent neural network) backtrack solver for better future stock prediction. This paper also explains the need for hybridizing evolutionary model CDRANN to create a novel change in future prediction.KeywordsCDRANN (Comprehensive Deep Recurrent Artificial Neural Networks)LSTM (long short-term memory)RNN (recurrent neural network)Stock marketPrediction

The reliability of Industrial Control Systems (ICS) is crucial given their substantial importance in society and business. These systems are susceptible to physical and cyber-attacks, which can seriously affect human life and the economy. Given the growing prevalence of Internet of Things (IoT) technologies and the possibility of cyber warfare, it is essential to prioritize safeguarding Industrial Control Systems (ICS) from cyberattacks. Previous studies revealed an imbalance in the ICS datasets. As a result, models didn't do well in minority classes but did well in majority classes, which made the intrusion detection system (IDS) less useful. The key objective is to provide insights into the normal functioning of the system and the disruptions produced by cyberattacks. In this study, we introduced an interdisciplinary framework that aims to enhance network intrusion detection systems (NIDSs). In this framework, we introduced an IDS via feature selection and feature reduction technique(s) with the attention-driven lightweight deep neural networks: Deep Recurrent Neural Networks (RNN), Deep Long Short-Term Memory (LSTM), and Deep Bi-directional Long Short-Term Memory (Bi-LSTM). Several feature selection techniques exclude features that fail to match the specified criteria. We employed Sparse Principal Component Analysis (SPCA) to extract higher-order features. We conducted experiments on the three datasets: the Secure Water Treatment System (SWaT), the Water Distribution (WADI), and the Gas Heating Loop (GHL).Among the models used in the framework, the attention-driven Deep LSTM model produced better results than the others, revealing lower training and testing times for the three datasets. In terms of precision, recall, F1-score, computational speed, and ability to work with larger datasets and different ICSs, the proposed framework is better than previous methods in detecting cyberattacks. This highlights how useful it is in the real world.

A novel anomaly detection method for multivariate time series based on spatial-temporal graph learning

Data available on the internet is growing constantly. This huge amount of data makes it harder for the users to get useful information. Comments and reviews of movies are given by many users, and recommendation systems makes it easier to find useful content, which is fast and relevant for users. Movies with more positive reviews and comments are usually chosen by everyone. The sentiment behind user reviews is useful to know whether a movie is worth watching. This paper outlines an approach for a movie recommendation system that uses cosine similarity technique to recommend similar movies to users based on the movie title , genre, director, actor they choose or search . Deep learning algorithms, GRU (Gated Recurrent unit), LSTM (long short-term memory), RNN (Recurrent neural network) and BI-LSTM (bi-directional long short-term memory), are trained and tested to classify user comments taken from YouTube into positive, negative, and neutral sentiments. Accuracy, F1-score, precision, recall measures are utilized to assess the model from every perspective. After comparing four algorithms, as BI-LSTM outperformed other algorithms with an accuracy, recall, F1 score and precision, of 98.58%, 97.99% 98.26% and 98.54% respectively, Bi-LSTM (bi-directional long short-term memory) is used for performing sentiment analysis on the movie reviews dataset to understand users sentiment.

Recurrent neural networks (RNNs) are naturally suitable for speech recognition because of their ability of utilizing dynamically changing temporal information. Deep RNNs have been argued to be able to model temporal relationships at different time granularities, but suffer vanishing gradient problems. In this paper, we extend stacked long short-term memory (LSTM) RNNs by using grid LSTM blocks that formulate computation along not only the temporal dimension, but also the depth dimension, in order to alleviate this issue. Moreover, we prioritize the depth dimension over the temporal one to provide the depth dimension more updated information, since the output from it will be used for classification. We call this model the prioritized Grid LSTM (pGLSTM). Extensive experiments on four large datasets (AMI, HKUST, GALE, and MGB) indicate that the pGLSTM outperforms alternative deep LSTM models, beating stacked LSTMs with 4% to 7% relative improvement, and achieve new benchmarks among uni-directional models on all datasets.

Securing personal authentication is an important study in the field of security. Particularly, fingerprinting and face recognition have been used for personal authentication. However, these systems suffer from certain issues, such as fingerprinting forgery, or environmental obstacles. To address forgery or spoofing identification problems, various approaches have been considered, including electrocardiogram (ECG). For ECG identification, linear discriminant analysis (LDA), support vector machine (SVM), principal component analysis (PCA), deep recurrent neural network (DRNN), and recurrent neural network (RNN) have been conventionally used. Certain studies have shown that the RNN model yields the best performance in ECG identification as compared with the other models. However, these methods require a lengthy input signal for high accuracy. Thus, these methods may not be applied to a real-time system. In this study, we propose using bidirectional long short-term memory (LSTM)-based deep recurrent neural networks (DRNN) through late-fusion to develop a real-time system for ECG-based biometrics identification and classification. We suggest a preprocessing procedure for the quick identification and noise reduction, such as a derivative filter, moving average filter, and normalization. We experimentally evaluated the proposed method using two public datasets: MIT-BIH Normal Sinus Rhythm (NSRDB) and MIT-BIH Arrhythmia (MITDB). The proposed LSTM-based DRNN model shows that in NSRDB, the overall precision was 100%, recall was 100%, accuracy was 100%, and F1-score was 1. For MITDB, the overall precision was 99.8%, recall was 99.8%, accuracy was 99.8%, and F1-score was 0.99. Our experiments demonstrate that the proposed model achieves an overall higher classification accuracy and efficiency compared with the conventional LSTM approach.

Critical industrial infrastructures are currently facing increasing cyberspace threats in their underlying information and communication systems. The advanced monitoring, control, and management functionalities of the industrial systems firmly rely on the reliable and secure operations of the industrial control system (ICS) network. This article characterizes the ICS network traffic and presents a scalable and efficient solution for real-time ICS network traffic anomaly detection, considering various forms of ICS anomaly events. The events due to the cyberattacks, malicious operating behaviors, and network anomalies can be effectively detected without sophisticated computational requirements and retrieval of communication protocols. The proposed hybrid statistical-machine learning model integrates a seasonal autoregressive integration moving average (SARIMA)-based dynamic threshold model and a long short-term memory (LSTM) model to jointly identify the abnormal traffic patterns with low false omission rates. The proposed solution is extensively evaluated at a realistic ICS cyber–physical system (CPS) testbed, and the numerical results confirm its high detection accuracy and low computational complexity. Note to Practitioners—This article was motivated by the challenge of real-time anomaly detection in industrial cyber–physical systems (CPSs). The existing industrial control system (ICS) network anomaly detection solutions are generally carried out based on a single model based on the historian database and cannot dynamically classify the abnormal conditions in a real-time fashion. A novel hybrid statistical-machine learning model is developed that integrates a seasonal autoregressive integration moving average (SARIMA)-based dynamic threshold model and a long short-term memory (LSTM) model to jointly identify the anomalous events through traffic pattern analysis. The proposed anomaly detection solution can efficiently provide accurate detection for cyberattacks, malicious operating behaviors, and network anomalies while meeting the real-time requirements of ICS networks. The proposed solution can be deployed in the realistic ICS CPSs, e.g., the power generation system, gas pipeline systems, and urban railway transportation systems. The preliminary numerical results obtained from the ICS-CPS testbed suggested that it can provide high detection accuracy with low computational complexity and, hence, can be adopted with minimal deployment hurdles.

Most Graph Neural Networks (GNNs) follow the message-passing scheme. Residual connection is an effective strategy to tackle GNNs' over-smoothing issue and performance reduction issue on non-homophilic networks. Unfortunately, the coarse-grained residual connection still suffers from class-imbalanced over-smoothing issue, due to the fixed and linear combination of topology and attribute in node representation learning. To make the combination flexible to capture complicated relationship, this paper reveals that the residual connection needs to be node-dependent, layer-dependent, and related to both topology and attribute. To alleviate the difficulty in specifying complicated relationship, this paper presents a novel perspective on GNNs, i.e., the representations of one node in different layers can be seen as a sequence of states. From this perspective, existing residual connections are not flexible enough for sequence modeling. Therefore, a novel node-dependent residual connection, i.e., Long Short-Term Graph Memory Network (LSTGM) is proposed to employ Long Short-Term Memory (LSTM), to model the sequence of node representation. To make the graph topology fully employed, LSTGM innovatively enhances the updated memory and three gates with graph topology. A speedup version is also proposed for effective training. Experimental evaluations on real-world datasets demonstrate their effectiveness in preventing over-smoothing issue and handling networks with heterophily.

SummaryIn recent years, wireless sensor networks are pervasive and are generating tons of data every second. Performing outlier detection to detect faulty sensors from such a large amount of data becomes a challenging task. Most of the existing techniques for outlier detection in wireless sensor networks concentrate only on contents of the data source without considering correlation among different data attributes. Moreover, these methods are not scalable to big data. To address these 2 limitations, this paper proposes an outlier detection approach based on correlation and dynamic SMO (sequential minimal optimization) regression that is scalable to big data. Initially, correlation is used to find out strongly correlated attributes and then the point anomalous nodes are detected using dynamic SMO regression. For fast processing of big data, Hadoop MapReduce framework is used. The experimental analysis demonstrates that the proposed approach efficiently detects the point and contextual anomalies and reduces the number of false alarms. For experiments, real data of sensors used in body sensor networks are taken from Physionet database.

Token-Based Adaptive Time-Series Prediction by Ensembling Linear and Non-linear Estimators: A Machine Learning Approach for Predictive Analytics on big Stock Data

The research study is to predict human activity recognition and analyze their strength in machine learning (ML) processes to achieve precision and also to determine performance of human activities. The framework to classify the Novel Long Short Term Memory (LSTM) and Decision Tree (DT) to perform all measures. This research study used the Novel Long Short Term Memory and Decision Tree to perform the required actions to accurately identify body movements in adjacent film sequence. Recent study findings and a threshold of 0.05%, as well as 95% confidence interval and standard parameter estimates, were castoff in a study 47 samples divided into two groups calculated using the G-power of 80%. Since the novel Long Short Term Memory (LSTM) algorithm and local invariant methods have achieved an accuracy of 93.83% in predicting the activity analysis, this research needs to obtain improved accuracy to perform activity prediction by using the Decision Tree algorithm. When applied to human activity analysis, the Novel Long Short Term Memory technique was shown to have an accuracy of 90.12 percent, with a 95% confidence interval including the value 0.045 (p 0.05). Researchers found that when comparing two algorithms for analyzing human behaviour, Long Short Term Memory (LSTM) performed much better than Decision Tree (DT).

Co-LSTM: Convolutional LSTM model for sentiment analysis in social big data