GRU-Based Interpretable Multivariate Time Series Anomaly Detection in Industrial Control System

PdGAT-ID: An intrusion detection method for industrial control systems based on periodic extraction and spatiotemporal graph attention

The development and integration of information technology and industrial control networks have expanded the magnitude of new data; detecting anomalies or discovering other valid information from them is of vital importance to the stable operation of industrial control systems. This paper proposes an incremental unsupervised anomaly detection method that can quickly analyze and process large-scale real-time data. Our evaluation on the Secure Water Treatment dataset shows that the method is converging to its offline counterpart for infinitely growing data streams.

The classification of multidimensional time series holds significant importance across various domains, including action classification, medical diagnosis, and credit assessment. Within multidimensional time series data, features pertinent to classification exhibit variance in their positional distribution along the entirety of the sequence. Moreover, the relative significance of features across distinct dimensions also fluctuates, contributing to suboptimal performance in multidimensional time series classification. Consequently, the proposition of tailored deep learning models for feature extraction specific to multidimensional time series data becomes imperative. This paper introduces attention mechanisms applied to the temporal dimension, graph attention mechanisms for inter-dimensional relationships within multidimensional data, and attention mechanisms applied between channels post-convolutional calculations. These mechanisms are deployed for feature extraction across temporal, variational, and channel dimensions of multidimensional time series data, respectively. Furthermore, attention is directed towards inter-channel interactions within the squeeze-and-excitation network to enhance the model’s representational capacity. Experimental findings substantiate the viability of integrating attention mechanisms into multidimensional time series classification endeavors.

BackgroundThe prediction of coronavirus disease in 2019 (COVID-19) in broader regions has been widely researched, but for specific areas such as urban areas the predictive models were rarely studied. It may be inaccurate to apply predictive models from a broad region directly to a small area. This paper builds a prediction approach for small size COVID-19 time series in a city.MethodsNumbers of COVID-19 daily confirmed cases were collected from November 1, 2022 to November 16, 2023 in Xuzhou city of China. Classical deep learning models including recurrent neural network (RNN), long and short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN) are initially trained, then RNN, LSTM and GRU are integrated with a new attention mechanism and transfer learning to improve the performance. Ten times ablation experiments are conducted to show the robustness of the performance in prediction. The performances among the models are compared by the mean absolute error, root mean square error and coefficient of determination.ResultsLSTM outperforms than others, and TCN has the worst generalization ability. Thus, LSTM is integrated with the new attention mechanism to construct an LSTMATT model, which improves the performance. LSTMATT is trained on the smoothed time series curve through frequency domain convolution augmentation, then transfer learning is adopted to transfer the learned features back to the original time series resulting in a TLLA model that further improves the performance. RNN and GRU are also integrated with the attention mechanism and transfer learning and their performances are also improved, but TLLA still performs best.ConclusionsThe TLLA model has the best prediction performance for the time series of COVID-19 daily confirmed cases, and the new attention mechanism and transfer learning contribute to improve the prediction performance in the flatten part and the jagged part, respectively.

With the continuous progress of the society, the demand for electrical power is urgent. The transformer plays an important role in the power energy transmission. The oil temperature inside transformer effectively could reflect working condition of the transformer, which makes it necessary to monitor and forecast the oil temperature to monitor the operating status of the power transformer. However, the oil temperature time series data generated by the power transformer has the characteristics of being complex and nonlinear. In recent years, long and short time memory networks (LSTM) are often used to predict transformer oil temperature. Gated recurrent unit (GRU) is a new version for LSTM. In the structure of GRU, there exist two gates, which are updating gate and resetting gate, respectively. Compared with LSTM network, The structure of GRU is simpler and its effect is better. A novel predicting method for transformer oil temperature is proposed based on time series theory and GRU in this paper, which is verified on the dataset of the oil temperature of the transformers in the two regions. The experimental results are compared with traditional time series prediction models to demonstrate that the proposed method is effective and feasible.

Speech signals carry various bits of information relevant to the speaker such as age, gender, accent, language, health, and emotions. Emotions are conveyed through modulations of facial and vocal expressions. This paper conducts an empirical comparison of performances between the classical classifiers: Gaussian Mixture Model (GMM), Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Artificial neural networks (ANN); and the deep learning classifiers, i.e., Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Gated Recurrent Unit (GRU) in addition to the ivector approach for a text-independent speaker verification task in neutral and emotional talking environments. The deep models undergo hyperparameter tuning using the Grid Search optimization algorithm. The models are trained and tested using a private Arabic Emirati Speech Database, Ryerson Audio–Visual Database of Emotional Speech and Song dataset (RAVDESS) database, and a public Crowd-Sourced Emotional Multimodal Actors (CREMA) database. Experimental results illustrate that deep architectures do not necessarily outperform classical classifiers. In fact, evaluation was carried out through Equal Error Rate (EER) along with Area Under the Curve (AUC) scores. The findings reveal that the GMM model yields the lowest EER values and the best AUC scores across all datasets, amongst classical classifiers. In addition, the ivector model surpasses all the fine-tuned deep models (CNN, LSTM, and GRU) based on both evaluation metrics in the neutral, as well as the emotional speech. In addition, the GMM outperforms the ivector using the Emirati and RAVDESS databases.

Continuous annealing transition state evaluation of ultra-high strength steel via deep clustering network with time-series data

This research offers an in-depth examination of predicting the closing prices of the metal ores industry index on the Tehran Stock Exchange (TSE) using a Gated Recurrent Unit (GRU) model. The GRU, a type of recurrent neural network, shows great promise for tasks involving time series forecasting. The historical daily price data from October 2017 to October 2022, was used in the study after carefully preprocessing it for further analysis. The study begins with a univariate analysis to reveal distribution characteristics and the relationships between essential variables. A customized GRU model that is trained on 70% of the time series data, with its performance assessed through metrics such as Root Mean Square Error (RMSE), Mean Square Error (MSE), Mean Absolute Error (MAE), and the R-squared (R2) score is used for prediction. The results indicate that the GRU model provides accurate predictions for the metal ores industry index, outperforming traditional forecasting techniques. The model's recurrent nature enables it to capture both short-term and long-term temporal dependencies within the data. This research highlights the significant potential of GRU networks in the realm of financial forecasting. Future improvements will focus on hyperparameter optimization and further integrating additional input variables to enhance predictive accuracy.

Time series shapelets are small subsequences that maximally differentiate classes of time series. Since the inception of shapelets, researchers have used shapelets for various data domains including anthropology and health care, and in the process suggested many efficient techniques for shapelet discovery. However, multi-dimensional time series data poses unique challenges to shapelet discovery that are yet to be solved. We show that an ensemble of shapelet-based decision trees on individual dimensions works better than shapelets defined over multiple dimensions. Generating a shapelet ensemble for multidimensional time series is computationally expensive. Most of the existing techniques prune shapelet candidates for speed. In this paper, we propose a novel technique for shapelet discovery that evaluates remaining candidates efficiently. Our algorithm uses a multi-length approximate index for time series data to efficiently find the nearest neighbors of the candidate shapelets. We employ a simple skipping technique for additional candidate pruning and a voting based technique to improve accuracy while retaining interpretability. Not only do we find a significant speed increase, our techniques enable us to efficiently discover shapelets on datasets with multi-dimensional and long time series such as hours of brain activity recordings. We demonstrate our approach on a biomedical dataset and find significant differences between patients with schizophrenia and healthy controls.

Evaluating the Impact of Cyberattacks on PLC Performance: A Systematic Implementation and Empirical Investigation

The rapid development of technology during the last decades has led to the integration of the network capabilities in the devices that are essential in the operation of Industrial Control Systems (ICS). Consequently, the attack surface of these assets has increased, putting at risk the nations that depend heavily on them for their continuity and functions. ICS physical and virtual testbeds are ideal platforms that can be used for cybersecurity research. Compared to physical testbeds, virtual testbeds are less expensive, safer to maintain, and more accessible, especially when access to research labs is restricted due to unforeseen circumstances such as COVID 19. Therefore, in this article, we propose VNWTS, a virtual water chlorination process for ICS cybersecurity analysis. VNWTS is composed of virtual components commonly found in physical ICS implementations. In addition, we implemented a set of attacks targeting the memory of the S7-1500 PLC and other VNWTS components such as level sensors, temperature sensors, and pumps. The results obtained from the experimentation phase show that the structure, configuration, and implementation of the VNWTS testbed can be used in ICS cybersecurity research.

Hepatocellular carcinoma (HCC) remains a significant global health concern, often diagnosed at advanced stages, limiting the efficacy of surgical interventions. Early and accurate diagnosis is critical for improving surgical outcomes and reducing mortality. Traditional biomarkers, such as alpha-fetoprotein (AFP), des-gamma-carboxyprothrombin (DCP), and the lectin-bound fraction of AFP (AFP-L3), show limited sensitivity and specificity. Advanced diagnostic models, including GALAD, TAGALAD, and GAP_TALAD, offer a promising multi-biomarker approach but lack extensive evaluation in surgical contexts. This retrospective study included a cohort of 267 untreated hepatocellular carcinoma patients and 231 control patients (with hepatitis or cirrhosis). We applied the predefined formulas for the TAGALAD, GAP_TALAD, and other models to the cohort data. The diagnostic performance of each model and individual biomarker for detecting HCC was assessed using receiver operating characteristic (ROC) curve analysis to determine the area under the curve (AUC), sensitivity, and specificity at optimal cut-offs. Additionally, key clinical subgroups, including pathologically confirmed HCC, clinically diagnosed HCC, early-stage HCC (TNM I+II), patients with complete data (no imputation), and hepatitis B virus (HBV)-related disease, were also analyzed. TAGALAD and GAP_TALAD demonstrated superior performance compared to the GALAD model and traditional biomarkers across all patient subgroups. Notably, TAGALAD achieved the highest diagnostic accuracy, with an AUC of 0.880, sensitivity of 0.760, and specificity of 0.861, followed closely by GAP_TALAD (AUC = 0.874). Both models demonstrated excellent performance in early-stage HCC detection (TAGALAD AUC = 0.860, GAP_TALAD AUC = 0.867), highlighting their potential in identifying candidates for surgical resection or transplant at an early curative stage. In HBV-related HCC, TAGALAD (AUC = 0.874) and GAP_TALAD (AUC = 0.857) showed superior diagnostic accuracy compared to GALAD (AUC = 0.731) and single biomarkers (AUC = 0.598-0.799). The TAGALAD and GAP_TALAD models offer a robust and reliable framework that supports early diagnosis of HCC. Their superior accuracy indicates a more reliable foundation for identifying candidates for curative surgical interventions, suggesting the potential to refine patient selection. Future research should focus on multi-center validation and the integration of novel biomarkers to further optimize these models for surgical decision-making and personalized treatment strategies.

Prediction of dissolved oxygen in a fishery pond based on gated recurrent unit (GRU)

In the field of network intrusion detection based on a neural network with Gated Recurrent Units (GRU), the accuracy of classification decreases due to the improper setting of the parameters. Therefore an Improved Seagull Optimization Algorithm (ISOA) to optimize GRU parameters has been proposed. The ISOA is proposed to address the shortcomings of the Seagull Optimization Algorithm (SOA) of low convergence accuracy and slow convergence speed in dealing with global optimization problems. The algorithm uses an opposition-based learning method instead of a random method to initialize the population so that the population has a better diversity. To balance the overall and local search capability of the algorithm, in this paper, we design a special factor update formula that converges nonlinearly with the number of iterations, allowing the algorithm to better jump out of locally optimal solutions and speed up convergence. The improved algorithm is used to obtain better parameters and uses them to set the parameters of the GRU to build a better-performing classification model. Multiple UCI datasets are used and compared with other methods and finally validated using the NSL-KDD datasets for network intrusion detection. The experimental results show that compared to the performance of long short-term memory(LSTM) units, multilayer perceptron(MLP), and the traditional neural network with GRU, the classification accuracy of the method with ISOA for optimizing GRU is improved on each dataset, which helps to improve the classification performance in network intrusion detection.

In recent years, channel state information (CSI) in WiFi 802.11n has been increasingly used to collect data pertaining to human activity. Such raw data are then used to enhance human activity recognition. Activities such as lying down, falling, walking, running, sitting down, and standing up can now be detected with the use of information collected through CSI. Human activity recognition has a multitude of applications, such as home monitoring of patients. Four deep learning models are presented in this paper, namely: a convolution neural network (CNN) with a Gated Recurrent Unit (GRU); a CNN with a GRU and attention; a CNN with a GRU and a second CNN, and a CNN with Long Short-Term Memory (LSTM) and a second CNN. Those models were trained to perform Human Activity Recognition (HAR) using CSI amplitude data collected by a CSI tool. Experiments conducted to test the efficacy of these models showed superior results compared with other recent approaches. This enhanced performance of our models may be attributable the ability of our models to make full use of available data and to extract all data features, including high dimensionality and time sequence. The highest average recognition accuracy reached by the proposed models was achieved by the CNN-GRU, and the CNN-GRU with attention models, standing at 99.31% and 99.16%, respectively. In addition, the performance of the models was evaluated for unseen CSI data by training our models using a random split-of-dataset method (70% training and 30% testing). Our models achieved impressive results with accuracies reaching 100% for nearly all activities. For the lying down activity, accuracy obtained from the CNN-GRU model stood at 99.46%; slightly higher than the 99.05% achieved by the CNN-GRU with attention model. This confirmed the robustness of our models against environmental changes.