TSDR-SFE: A prediction model for dam crack width based on two-stage decomposition–reconstruction and spatiotemporal feature extraction

A hybrid prediction model of air quality for sparse station based on spatio-temporal feature extraction

Neuromorphic speech recognition systems that use spiking neural networks (SNNs) and memristors are progressing in hardware development. The conventional manual preprocessing of audio signals is shifting toward event-based recognition with convolutional SNNs. Despite achieving high accuracy in classification, the efficient extraction of spatiotemporal features from audio events continues to be a substantial challenge. In this study, we introduce dynamic time-surface neurons (DTSNs) using volatile memristors featuring an adjustable temporal kernel decay, enabled by series-connected transistors with an Au/LiCoO 2 /Au configuration. DTSNs act as feature descriptors, enhancing the spatiotemporal feature extraction from event audio data. A two-layer SNN classifier, fully connected and incorporating a 1T1R nonvolatile memristor array, is trained to recognize the spatiotemporal features of the audio data. Our findings show classification accuracies of up to 95.91%, substantial improvements in computational efficiency, and increased noise resilience, confirming the promise of our memristor-based speech recognition system for practical applications.

The development of cloud computing and virtualization technology has brought great challenges to the reliability of data center services. Data centers typically contain a large number of compute and storage nodes which may fail and affect the quality of service. Failure prediction is an important means of ensuring service availability. Predicting node failure in cloud-based data centers is challenging because the failure symptoms reflected have complex characteristics, and the distribution imbalance between the failure sample and the normal sample is widespread, resulting in inaccurate failure prediction. Targeting these challenges, this paper proposes a novel failure prediction method FP-STE (Failure Prediction based on Spatio-temporal Feature Extraction). Firstly, an improved recurrent neural network HW-GRU (Improved GRU based on HighWay network) and a convolutional neural network CNN are used to extract the temporal features and spatial features of multivariate data respectively to increase the discrimination of different types of failure symptoms which improves the accuracy of prediction. Then the intermediate results of the two models are added as features into SCSXGBoost to predict the possibility and the precise type of node failure in the future. SCS-XGBoost is an ensemble learning model that is improved by the integrated strategy of oversampling and cost-sensitive learning. Experimental results based on real data sets confirm the effectiveness and superiority of FP-STE.

The demand for encrypted communication is increasing with the continuous development of secure and trustworthy networks. In edge computing scenarios, the requirement for data processing security is becoming increasingly high. Therefore, the accurate identification of encrypted traffic has become a prerequisite to ensure edge intelligent device security. Currently, encrypted network traffic classification relies on single-feature extraction methods. These methods have simple feature extraction, making distinguishing encrypted network data flows and designing compelling manual features challenging. This leads to low accuracy in multi-classification tasks involving encrypted network traffic. This paper proposes a hybrid deep learning model for multi-classification tasks to address this issue based on the synergy of dilated convolution and gating unit mechanisms. The model comprises a Gated Dilated Convolution (GDC) module and a CA-LSTM module. The GDC module completes the spatial feature extraction of encrypted network traffic through dilated convolution and gating unit mechanisms. In contrast, the CA-LSTM module focuses on extracting temporal network traffic features. By employing a collaborative approach to extract spatio-temporal features, the model ensures feature extraction diversity, guarantees robustness, and effectively enhances the feature extraction rate. We evaluate our multi-classification model using the ISCX VPN-nonVPN public dataset. Experimental results show that the proposed method achieves an accuracy rate of over 95% and a recall rate of over 90%, significantly outperforming existing methods.

This study proposes a feature extraction scheme that fuses accelerometric (ACC) and electromyographic (EMG) data to improve shoulder movement identification in individuals with transhumeral amputation, in whom the clinical need for intuitive control strategies enabling reliable activation of full-arm prostheses is underinvestigated. A novel spatiotemporal warping feature extraction architecture was employed to realize EMG and ACC information fusion at the feature level. EMG and ACC data were collected from six participants with intact limbs and four participants with transhumeral amputation using an NI USB-6009 device at 1000 Hz to support the proposed feature extraction scheme. For each participant, a leave-one-trial-out (LOTO) training and testing approach was used for developing pattern recognition models for both the intact-limb (IL) and amputee (AMP) groups. The analysis revealed that the introduction of ACC information has a positive impact when using windows of length (WLs) lower than 150 ms. A linear discriminant analysis (LDA) classifier was able to exceed the accuracy of 90% in each WL condition and for each group. Similar results were observed for an extreme learning machine (ELM), whereas k-nearest neighbors (kNN) and an autonomous learning multi-model classifier showed a mean accuracy of less than 87% for both IL and AMP groups at different WLs, guaranteeing applicability over a large set of shallow pattern-recognition models that can be used in real scenarios. The present work lays the groundwork for future studies involving real-time validation of the proposed methodology on a larger population, acknowledging the current limitation of offline analysis.

Video captioning is a technique used to represent videos using phrases of natural language. This technique automatically generates phrases to explain the contents of input videos. Deep learning-based video captioning methodologies are used for generating descriptions in natural language for a given video or an image. The significance of video captioning in the present era has advanced research in this field during the last few decades. A methodical review of recent works, various benchmarks used for performance evaluation, and various data sets for training are explained in this review paper for the benefit of new researchers. The most widely used neural network variants for visual and spatiotemporal feature extraction, as well as an approach for language generation, are included in this survey paper. The results show that ResNet and VGG are the most widely used visual feature extractors, and 3D convolutional neural networks are the most widely used Spatio-temporal feature extractors. Aside from that, the language model has primarily been used as Long Short-Term Memory (LSTM). The most widely used metrics for video description evaluation are BLEU, ROUGE, METEOR, CIDEr, SPICE, and WMD. This article is intended to be an extensive overview of deep learning-based captioning approaches for a given image and video.

Reciprocating compressors, which serve as core equipment in the petrochemical and natural gas transmission sectors, operate under prolonged variable loads and high-frequency impact conditions. Critical components, such as valves and piston rings, are prone to failure. Existing fault diagnosis methods suffer from inadequate spatio-temporal feature extraction and neglect spatio-temporal correlations. To address this, this paper proposes a spatio-temporal feature fusion-based fault diagnosis method for reciprocating compressors. This method constructs a spatio-temporal feature fusion model (STFFM) comprising three principal modules: First, a spatio-temporal feature extraction module employing a multi-layered stacked bidirectional gated recurrent unit (BiGRU) with batch normalisation to uncover temporal dependencies in long-term sequence data. A graph structure is constructed via k-nearest neighbours (KNN), and an enhanced graph isomorphism network (GIN) is integrated to capture spatial domain fault information variations. Second, the spatio-temporal bidirectional attention-gated fusion module employs a bidirectional multi-head attention mechanism to enhance temporal and spatial features. It incorporates a cross-modal gated update mechanism and learnable weight parameters to dynamically retain the highly discriminative features. Third, the classification output module enhances the model’s generalisation capability through multi-layer fully connected layers and regularisation design. Research findings demonstrate that this approach effectively integrates spatio-temporal coupled fault features, achieving an average accuracy of 99.14% on an experimental dataset. This provides an effective technical pathway for the precise identification of faults in the critical components of reciprocating compressors.

At present, the amount of network equipment, servers, and network traffic is increasing exponentially, and the way in which operators allocate and efficiently utilize network resources has attracted considerable attention from traffic forecasting researchers. However, with the advent of the 5G era, network traffic has also shown explosive growth, and network complexity has increased dramatically. Accurately predicting network traffic has become a pressing issue that must be addressed. In this paper, a multilayer perceptron ensemble learning method based on convolutional neural networks (CNN) and gated recurrent units (GRU) spatiotemporal feature extraction (MECG) is proposed for network traffic prediction. First, we extract spatial and temporal features of the data by convolutional neural networks (CNN) and recurrent neural networks (RNN). Then, the extracted temporal features and spatial features are fused into new spatiotemporal features through integrated learning of a multilayer perceptron, and a spatiotemporal prediction model is built in the sequence-to-sequence framework. At the same time, the teacher forcing mechanism and attention mechanism are added to improve the accuracy and convergence speed of the model. Finally, the proposed method is compared with other deep learning models for experiments. The experimental results show that the proposed method not only has apparent advantages in accuracy but also shows some superiority in time training cost.

Satellite Image Time Series (SITS), MRI sequences, and more generally image time series, constitute $2D+t$ data providing spatial and temporal information about an observed scene. Given a pattern recognition task such as image classification, considering jointly such rich information is crucial during the decision process. Nevertheless, due to the complex representation of the data-cube, spatio-temporal features extraction from $2D+t$ data remains difficult to handle. We present in this article an approach to learn such features from this data, and then to proceed to their classification. Our strategy consists in enriching pixel time series with spatial information. It is based on Random Walk to build a novel segment-based representation of the data, passing from a $2D+t$ dimension to a $2D$ one, without loosing too much spatial information. Such new representation is then involved in an end-to-end learning process with a classical 2D Convolutional Neural Network (CNN) in order to learn spatiotemporal features for the classification of image time series. Our approach is evaluated on a remote sensing application for the mapping of agricultural crops. Thanks to a visual attention mechanism, the proposed $2D$ spatio-temporal representation makes also easier the interpretation of a SITS to understand spatiotemporal phenomenons related to soil management practices.

Remote photoplethysmography (rPPG) is a non-contact technology that estimates physiological signals, such as Heart Rate (HR), by capturing subtle skin color changes caused by periodic blood volume variations using only a standard RGB camera. While cost-effective and convenient, it suffers from a fundamental limitation: performance degrades severely in dynamic environments due to susceptibility to noise, such as abrupt illumination changes or motion blur. This study presents a deep learning framework that combines two structural modifications to ensure robustness in dynamic environments, specifically modeling movement noise and illumination change noise. The proposed framework structurally cancels global disturbances, such as illumination changes or global motion, through a dual-branch pipeline that encodes the face and background in parallel after Video Color Magnification (VCM) and then performs differencing. Subsequently, it utilizes a structure that injects a Temporal Shift Module (TSM) into the Spatio-Temporal Feature Extraction (SSFE) block to preserve long- and short-term temporal correlations and smooth noise, even amidst short and irregular movements. We measured MAE, RMSE, and correlation on the standard dataset UBFC-rPPG under four noise conditions: clean, illumination change noise, Movement Noise, Both Noise and the real-world in-vehicle dataset MR-NIRP (Stationary and Driving). Experimental results showed that the proposed method achieved consistent error reduction and correlation improvement compared to the VS-Net baseline in the illumination change noise-only and combined noise environments (UBFC-rPPG) and in the high-noise driving scenario (MR-NIRP). It maintained competitive performance in motion-only noise. Conversely, a modest performance disadvantage was observed under clean conditions (UBFC) and quasi-clean stationary conditions (MR-NIRP), interpreted as a design trade-off focused on global noise cancellation and temporal smoothing. Ablation studies demonstrated that the dual-branch pipeline is the primary contributor under illumination change noise, while TSM is the key contributor under movement noise, and that the combination of both elements achieves optimal robustness in the most complex scenarios.

The total electron content (TEC) is a crucial factor for quantify the particle concentration in the ionosphere, significantly impacting the performance of global navigation satellite systems. Related researches on TEC prediction are therefore essential for ionospheric modeling and its applications. Existing approaches are generally limited in computational efficiency and feature extraction capability. To address these problems, a filter-integrated TEC prediction model with a learnable frequency kernel is proposed to extract key characteristics from the frequency domain perspective. Experimental results show that under different ionospheric disturbances, the proposed model achieves approximately 18% to 40% improvement in RMSE compared to Transformer, and 10% to 14% improvement compared to iTransformer. This method demonstrates superior performance on spatiotemporal feature extraction over the self-attention mechanisms, revealing its wide prospects in the areas of real-time ionospheric modeling and positioning delay correction.

Spatio-temporal feature extraction network based multi-performance indicators synergetic monitoring method for complex industrial processes

This Letter presents a stable polyp-scene classification method with low false positive (FP) detection. Precise automated polyp detection during colonoscopies is essential for preventing colon-cancer deaths. There is, therefore, a demand for a computer-assisted diagnosis (CAD) system for colonoscopies to assist colonoscopists. A high-performance CAD system with spatiotemporal feature extraction via a three-dimensional convolutional neural network (3D CNN) with a limited dataset achieved about 80% detection accuracy in actual colonoscopic videos. Consequently, further improvement of a 3D CNN with larger training data is feasible. However, the ratio between polyp and non-polyp scenes is quite imbalanced in a large colonoscopic video dataset. This imbalance leads to unstable polyp detection. To circumvent this, the authors propose an efficient and balanced learning technique for deep residual learning. The authors’ method randomly selects a subset of non-polyp scenes whose number is the same number of still images of polyp scenes at the beginning of each epoch of learning. Furthermore, they introduce post-processing for stable polyp-scene classification. This post-processing reduces the FPs that occur in the practical application of polyp-scene classification. They evaluate several residual networks with a large polyp-detection dataset consisting of 1027 colonoscopic videos. In the scene-level evaluation, their proposed method achieves stable polyp-scene classification with 0.86 sensitivity and 0.97 specificity.

Character-level arabic text generation from sign language video using encoder–decoder model

Obstructive sleep apnea (OSA) is a common sleep disorder that is closely associated with snoring patterns. Since acoustic monitoring provides a non-invasive and easy-to-use method for detecting sleep abnormalities, this study presents a YAMNet-based hierarchical attention framework for classifying nocturnal audio. The proposed method uses a pre-trained YAMNet architecture to automatically extract Mel-spectrograms features, and further converts the raw audio into a three-channel spectral representation. After the above processing, the features are fed into a classification network that integrates dual convolutional layers, an attention mechanism, and gated recurrent units for hierarchical feature processing. This two-stage process starts with the extraction of distinctive snores and then applies spatial channel attention and temporal modelling for accurate classification. Nocturnal sounds are classified into three categories: respiratory events, OSA-related snoring, and ambient noise. Experimental results show that the method achieves 96.7% accuracy in detecting snoring on the training set of data and 98.30% average accuracy on the test set, which is significantly better than traditional methods. This non-invasive monitoring tool can help patients understand their sleep health and support health management and disease prevention.