Effective Gait Feature Extraction Using Temporal Fusion And Spatial Partial

Recently, many deep learning based researches are conducted to explore the potential quality improvement of compressed videos. These methods mostly utilize either the spatial or temporal information to perform frame-level video enhancement. However, they fail in combining different spatial-temporal information to adaptively utilize adjacent patches to enhance the current patch and achieve limited enhancement performance especially on scene-changing and strong-motion videos. To overcome these limitations, we propose a patch-wise spatial-temporal quality enhancement network which firstly extracts spatial and temporal features, then recalibrates and fuses the obtained spatial and temporal features. Specifically, we design a temporal and spatial-wise attention-based feature distillation structure to adaptively utilize the adjacent patches for distilling patch-wise temporal features. For adaptively enhancing different patch with spatial and temporal information, a channel and spatial-wise attention fusion block is proposed to achieve patch-wise recalibration and fusion of spatial and temporal features. Experimental results demonstrate our network achieves peak signal-to-noise ratio improvement, 0.55 - 0.69 dB compared with the compressed videos at different quantization parameters, outperforming state-of-the-art approach.

Gait recognition is a distinctive biometric technique that can identify pedestrians by their walking patterns from considerable distances. A critical challenge in gait recognition lies in effectively acquiring discriminative spatial-temporal representations from silhouettes that exhibit invariance to disturbances. In this paper, we present a novel gait recognition network by aggregating features in the spatial-temporal and view domains, which consists of two-path spatial-temporal feature fusion module and view embedding module. Specifically, two-path spatial-temporal feature fusion module firstly utilizes multi-scale feature extraction (MSFE) to enrich the input features with multiple convolution kernels of various sizes. Then, frame-level spatial feature extraction (FLSFE) and multi-scale temporal feature extraction (MSTFE) are parallelly constructed to capture spatial and temporal gait features of different granularities and these features are fused together to obtain muti-scale spatial-temporal features. FLSFE is designed to extract both global and local gait features by employing a specially designed residual operation. Simultaneously, MSTFE is applied to adaptively interact multi-scale temporal features and produce suitable motion representations in temporal domain. Taking into account the view information, we introduce a view embedding module to reduce the impact of differing viewpoints. Through the extensive experimentation over CASIA-B and OU-MVLP datasets, the proposed method has achieved superior performance to the other state-of-the-art gait recognition approaches.

In autonomous driving and robotic navigation, multi-sensor fusion technology has become increasingly mainstream, with precise sensor calibration as its foundation. Traditional calibration methods rely on manual effort or specific targets, limiting adaptability to complex environments. Learning-based calibration methods still face challenges, such as insufficient overlap between the fields of view (FoV) of multiple sensors and suboptimal cross-modal feature association, which hinder accurate parameter regression. Unlike traditional CNN-based networks, we propose a KansFormer-based self-Calibration Network for camera and LiDAR (KFCalibNet) that replaces fixed activation functions and linear transformations with learnable nonlinear activation functions. This enables the extraction of more fine-grained features from both image and point cloud, significantly enhancing the network's robustness in scenarios with limited FoV overlap. We also employ a multihead attention (MHA) module to compute correlations between image and point cloud features, significantly enhancing cross-modal feature association. To reduce learning complexity, we designed KansFormer with FastKAN as the feedforward network, enabling deep fusion and regression of fine-grained cross-modal features for accurate extrinsic calibration. KFCalibNet achieves an absolute average calibration error of 0.0965 cm in translation and 0.0234° in rotation on the KITTI Odometry dataset, outperforming existing state-of-the-art calibration methods. Moreover, its accuracy and generalization capability have been validated across multiple real-world railway lines.

Graph convolutional network – Long short term memory neural network- multi layer perceptron- Gaussian progress regression model: A new deep learning model for predicting ozone concertation

In order to solve the problem of poor spatial and temporal feature extraction in traffic flow prediction, a multimodal spatial-temporal graph convolution network (MMSTGCN) is proposed. The road topology map is constructed, the spatial features are extracted by graph convolution network, and the fine-grained learning of node domain is added to extract the coarse-grained and fine-grained features. The time is divided into three kinds of time slices, and the time feature is extracted by gated cycle unit (GRU). Finally, three time slice features are fused for training. Two real world datasets, PeMSD4 and PeMSD8, are used to verify the effectiveness of the method.

Multi-view fusion neural network for traffic demand prediction

As a unique biometric feature that can be recognized at a distance, gait has broad applications in crime prevention, forensic identification and social security. To portray a gait, existing gait recognition methods utilize either a gait template, where temporal information is hard to preserve, or a gait sequence, which must keep unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper we present a novel perspective, where a gait is regarded as a set consisting of independent frames. We propose a new network named GaitSet to learn identity information from the set. Based on the set perspective, our method is immune to permutation of frames, and can naturally integrate frames from different videos which have been filmed under different scenarios, such as diverse viewing angles, different clothes/carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 95.0% on the CASIA-B gait dataset and an 87.1% accuracy on the OU-MVLP gait dataset. These results represent new state-of-the-art recognition accuracy. On various complex scenarios, our model exhibits a significant level of robustness. It achieves accuracies of 87.2% and 70.4% on CASIA-B under bag-carrying and coat-wearing walking conditions, respectively. These outperform the existing best methods by a large margin. The method presented can also achieve a satisfactory accuracy with a small number of frames in a test sample, e.g., 82.5% on CASIA-B with only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.

A network traffic prediction model based on reinforced staged feature interaction and fusion

Various techniques using artificial intelligence (AI) have resulted in a significant contribution to field of medical image and video-based diagnoses, such as radiology, pathology, and endoscopy, including the classification of gastrointestinal (GI) diseases. Most previous studies on the classification of GI diseases use only spatial features, which demonstrate low performance in the classification of multiple GI diseases. Although there are a few previous studies using temporal features based on a three-dimensional convolutional neural network, only a specific part of the GI tract was involved with the limited number of classes. To overcome these problems, we propose a comprehensive AI-based framework for the classification of multiple GI diseases by using endoscopic videos, which can simultaneously extract both spatial and temporal features to achieve better classification performance. Two different residual networks and a long short-term memory model are integrated in a cascaded mode to extract spatial and temporal features, respectively. Experiments were conducted on a combined dataset consisting of one of the largest endoscopic videos with 52,471 frames. The results demonstrate the effectiveness of the proposed classification framework for multi-GI diseases. The experimental results of the proposed model (97.057% area under the curve) demonstrate superior performance over the state-of-the-art methods and indicate its potential for clinical applications.

The LST-SATM-net: A new deep feature learning framework for aero-engine hydraulic pipeline systems intelligent faults diagnosis

Evaluating cognitive load using electroencephalogram (EEG) signals is a crucial research area in the field of Brain-Computer Interfaces (BCI). However, achieving high accuracy and generalization in feature extraction and classification for cognitive load assessment remains a challenge, primarily due to the low signal-to-noise ratio of EEG signals and the inter-individual variability in EEG data. In this study, we propose a novel deep learning architecture that integrates temporal information features and functional connectivity features to enhance EEG signal analysis. Functional connectivity features capture inter-channel information, while temporal features are extracted from continuous signal segments using a Long Short-Term Memory (LSTM) network enhanced with an attention mechanism. The fusion strategy combines these two information streams to leverage their complementary strengths, resulting in improved classification performance. We evaluated our architecture on two publicly available datasets, and the results demonstrate its robustness in cognitive load recognition. Achieving performance comparable to the best existing methods on two public datasets. Ablation studies further substantiate the contributions of each module, demonstrating the importance of combining temporal and functional connectivity features for optimal results. These findings underscore the robustness and versatility of the proposed approach, paving the way for more effective EEG-based BCI applications.

Recently, studies on driver inattention state recognition as an advanced mobility application technology are being actively conducted to prevent traffic accidents caused by driver drowsiness and distraction. The driver inattention state recognition system is a technology that recognizes drowsiness and distraction by using driver behavior, biosignals, and vehicle data characteristics. Existing driver drowsiness detection systems are wearable accessories that have partial occlusion of facial features and light scattering due to changes in internal and external lighting, which results in momentary image resolution degradation, making it difficult to recognize the driver’s condition. In this paper, we propose a transformer based spatial temporal fusion network (STFTransNet) that fuses multi-modality information for improved driver inattention state recognition in images where the driver’s face is partially occluded by wearing accessories and the instantaneous resolution is degraded due to light scattering from changes in lighting in a driving environment. The proposed STFTransNet consists of (i) a mediapipe face mesh-based facial landmark extraction process for facial feature extraction, (ii) an RCN-based two-stream cross-attention process for learning spatial features of driver face and body action images, (iii) a TCN-based temporal feature extraction process for learning temporal features of extracted features, and (iv) an ensemble of spatial and temporal features and a classification process to recognize the final driver state. As a result of the experiment, the proposed STFTransNet achieved an accuracy of 4.56% better than the existing VBFLLFA model in the NTHU-DDD public DB, 3.48% better than the existing InceptionV3 + HRNN model in the StateFarm public DB, and 3.78% better than the existing VBFLLFA model in the YawDD public DB. The proposed STFTransNet is designed as a two-stream network that can input the driver’s face and action images and solves the degradation in driver inattention state recognition performance due to partial facial feature occlusion and light blur through spatial feature and temporal feature fusion.

Many gait recognition methods first partition the human gait into N-parts and then combine them to establish part-based feature representations. Their gait recognition performance is often affected by partitioning strategies, which are empirically chosen in different datasets. However, we observe that strips as the basic component of parts are agnostic against different partitioning strategies. Motivated by this observation, we present a strip-based multi-level gait recognition network, named GaitStrip, to extract comprehensive gait information at different levels. To be specific, our high-level branch explores the context of gait sequences and our low-level one focuses on detailed posture changes. We introduce a novel StriP-Based feature extractor (SPB) to learn the strip-based feature representations by directly taking each strip of the human body as the basic unit. Moreover, we propose a novel multi-branch structure, called Enhanced Convolution Module (ECM), to extract different representations of gaits. ECM consists of the Spatial-Temporal feature extractor (ST), the Frame-Level feature extractor (FL) and SPB, and has two obvious advantages: First, each branch focuses on a specific representation, which can be used to improve the robustness of the network. Specifically, ST aims to extract spatial-temporal features of gait sequences, while FL is used to generate the feature representation of each frame. Second, the parameters of the ECM can be reduced in test by introducing a structural re-parameterization technique. Extensive experimental results demonstrate that our GaitStrip achieves state-of-the-art performance in both normal walking and complex conditions. The source code is published at https://github.com/M-Candy77/GaitStrip.

With the increasing demand for object detection accuracy in scenarios such as intelligent transportation and autonomous driving, methods that utilize point cloud and voxel features to achieve multi-modal feature fusion have become increasingly common in 3D object detection. However, existing methods often rely on inefficient linear fusion strategies during the multi-modal feature fusion process, which fails to adequately capture the dependencies between multi-source features, leading to insufficient feature integration. Additionally, during feature extraction, limitations in network architecture result in a lack of interaction between shallow and deep features, causing the loss of fine-grained feature information, which particularly affects the detection of small objects.To address these issues, we propose the Hybrid Point-Voxel-Transformer Network HPP-TNet , a two-stage object detection framework that integrates point and pillar features. Specifically, we design a fine-grained pillar feature extraction module CFPEM , which effectively alleviates the feature loss problem caused by voxel downsampling through shallow-deep feature interaction and lightweight attention design. Next, we develop a transformer-based multi-scale feature fusion module TMFFM , which dynamically achieves cross-modal associations through amulti-head attention mechanism, enhancing context-aware features and fully realizing multi-source feature fusion. Experiments on the KITTI dataset demonstrate that our proposed algorithm achieves competitive detection performance compared to several state-of-the-art methods, particularly in the Cyclist and Pedestrian categories. Our code will be open-sourced soon.

Air pollution is a critical global environmental issue, further exacerbated by rapid industrialization and urbanization. Accurate prediction of air pollutant concentrations is essential for effective pollution prevention and control measures. The complex nature of pollutant data is influenced by fluctuating meteorological conditions, diverse pollution sources, and propagation processes, underscores the crucial importance of the spatial and temporal feature extraction for accurately predicting air pollutant concentrations. To address the challenges of data redundancy and diminished long-term prediction accuracy observed in previous studies, this paper presents an innovative approach to predict air pollutant concentrations leveraging advanced data analysis and deep learning methods. The proposed approach, termed KSC-ConvLSTM, integrates the k-nearest neighbors (KNN) algorithm, spatio-temporal attention (STA) mechanism, the residual block, and convolutional long short-term memory (ConvLSTM) neural network. The KNN algorithm adaptively selects highly correlated neighboring domains, while the residual block, enhanced with the STA mechanism, extracts spatial features from the input data. ConvLSTM further processes the output from STA-ConvNet to capture high-dimensional temporal and spatial features. The effectiveness of the KSC-ConvLSTM approach was validated through predictions of PM2.5 concentrations in Beijing and its surrounding urban agglomeration. The experimental results indicate that the KSC-ConvLSTM approach outperforms benchmark approaches in single-step, multi-step, and trend prediction. It demonstrates superior fitting accuracy and predictive performance. Quantitatively, the proposed KSC-ConvLSTM approach reduces the root mean square error (RMSE) by 4.216–8.458 for prediction averages of 1–12 h of PM2.5 in Beijing, compared with the benchmark approach. The findings show that the KSC-ConvLSTM approach shows considerable potential for predicting, preventing, and controlling air pollution.