Multimodal activity recognition with local block CNN and attention-based spatial weighted CNN

Lane detection plays an important role in autonomous driving. For video instance lane detection, both global spatial and temporal information is significantly important. However, the global spatial features and the temporal features are not been well exploited in recent studies. In this work, we address the video instance lane detection task by capturing global context based on non-local attention network. Specifically, we designed a twin non-local attention network to extract long-range dependencies along the spatial and temporal dimensions, respectively. Meanwhile, the global spatial and temporal features can be adaptively fused by gating mechanisms for better results. In addition, the frame-similarity loss is proposed to further exploit the information of adjacent frames. The experimental results on the video instance lane detection (VIL-100) dataset verify that our method achieves better results compared with other comparison methods. Ablation experiments further demonstrate the effectiveness of each sub-module.

For hyperspectral image (HSI) classification, two branch networks generally use the convolution neural networks (CNNs) to extract the spatial features and the long short-term memory (LSTM) to learn the spectral features. However, CNN with a local kernel neglects the global properties of the whole HSI. LSTM doesn't consider the macroscopic and detailed information of spectra. In this paper, we propose a dual-view spectral and global spatial feature fusion network (DSGSF) to extract the spatial-spectral features for HSI classification, including a spatial subnetwork and a spectral subnetwork. In the spatial subnetwork, we propose a global spatial feature representation model based on the encoder-decoder structure with channel attention and spatial attention to learn the global spatial features. In the spectral subnetwork, we design a dual-view spectral feature aggregation model with view attention to learn the diversity of spectral features. By fusing the two subnetworks, we construct DSGSF to extract the spatial-spectral features of HSI with strong discriminating performance. Experimental results on three public datasets illustrate that the proposed method can achieve competitive results compared with the state-of-the-art methods. Code: https://github.com/RZWang-WH/DSGSF.

In recent years, the integration of convolutional neural networks (CNNs) and Transformers has significantly advanced hyperspectral image (HSI) classification by jointly capturing local and global features. However, most existing methods primarily focus on the fusion of spectral–spatial features while neglecting the complementary information contained in frequency domain features. To address this issue, we propose a spatial frequency cross-attention fusion network (SFCFNet) that jointly models spectral, spatial, and frequency-domain features for HSI classification. The framework consists of three core modules: first, the multiscale spectral spatial feature learning (MSSFL) module extracts joint spectral spatial features using multi-scale 3D and 2D convolutions. Next, the triple-branch representation module (TBRM) employs three branches to capture global spatial features of large-scale structures, local spatial features of fine grained textures, and multi-scale frequency features based on Haar wavelet decomposition, providing complementary multi domain representations for subsequent deep fusion. Finally, the dual-domain feature cross-attention fusion module (DFCF) achieves effective fusion of spatial structures and frequency domain textures, enhancing the model's ability to separate complex backgrounds from fine-grained targets and thereby improving classification performance. Compared with other methods, SFCFNet achieves higher overall accuracy (OA) on the Salinas, Houston2013, WHU-Hi-LongKou, and Xuzhou datasets, reaching 99.05%, 98.07%, 98.76%, and 98.18%, respectively.

SOR-TC: Self-attentive octave ResNet with temporal consistency for compressed video action recognition

Accurate precipitation estimation is essential in agricultural production, water resource management, and flood forecasting. However, high-precision precipitation data remain very hard to obtain due to the complex spatio-temporal distribution of precipitation. Most existing methods considering spatio-temporal correlations in precipitation rely on a convolutional neural network for spatial feature extraction. However, these methods are less efficient in capturing global spatial features due to the local receptive fields of convolutional operators. In this study, we designed a Self-LSTM cell structure capable of effectively capturing temporal and global spatial features. Based on this, a self-attention precipitation fusion model (SAPFM) is proposed. The results demonstrate that SAPFM outperforms basic models and the original precipitation products. SAPFM improves by 28.8% and 21.8% on the Kling-Gupta efficiency (KGE) and Correlation Coefficient (CC) compared to the best-performing precipitation product (GsMap), respectively. Additionally, SAPFM reduces the Root Mean Square Error (RMSE) by 12.5%.

Effectively utilizing multimodal information (e.g., heart rate and acceleration) is a promising way to achieve wearable sensor based human activity recognition (HAR). In this paper, an activity recognition approach MARCEL (Multimodal Activity Recognition with Classifier Ensemble) is proposed, which exploits the diversity of base classifiers to construct a good ensemble for multimodal HAR, and the diversity measure is obtained from both labeled and unlabeled data. MARCEL uses neural network (NN) as base classifiers to construct the HAR model, and the diversity of classifier ensemble is embedded in the error function of the model. In each iteration, the error of the model is decomposed and back-propagated to base classifiers. To ensure the overall accuracy of the model, the weights of base classifiers are learnt in the classifier fusion process with sparse group lasso. Extensive experiments show that MARCEL is able to yield a competitive HAR performance, and has its superiority on exploiting multimodal signals.

In this paper, we propose an end-to-end deep learning framework that could extract the global spatial-temporal features from the videos based on the NetVladframework optimized with data augmentation. Given keyframes extracted from the original videos, we have presented a three-step action recognition framework: the first step of our framework is given by a data augmentation based on central crop, random crop and keyframe scaling. Then the second step is given by a local feature descriptor of each frame with Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation, which enables the network to capture spatial and temporal features simultaneously. The third step is given by aggregating global features for action recognition through the new generalized “Vector of Locally Aggregated Descriptors” (NetVLAD) layer optimized with a novel pooling strategy, avoiding the misjudgment caused by local features. The whole framework is trained and fine-tuned via an end-to-end way. It's demonstrated that the performance of our framework outperforms the state-of-the-art algorithms on UCF10l dataset. The competitive results clearly reveal an efficient action recognition of high accuracy (up to 91.25%) in fast time (close to 1.8s), which will significantly improve performance and application of video action recognition.

Alzheimer's disease (AD) and Parkinson's disease (PD) are types of neurodegenerative diseases that affect the body and get worse over time. The cause of AD mainly involves the buildup of protein which are abnormal, issues with the immune reaction, death of neurons. Different from this, the death of the neurons that make dopamine leads to PD and causes both motor and non-motor problems. MRI images are used to provide an early and correct diagnosis to enable timely treatment planning and management of the disease. In this paper, a design of an AI-based deep learning framework is proposed for the classification of neurodegenerative disease based on the brain MRI data. The pipeline that we propose begins with data preparation including data augmentation using InceptionGAN for augmentation of the dataset and fixing of class imbalance issues. A composite method of feature extraction using ConvNeXt and MaxViT along with the Cross-Fusion Attention model, worked well to capture local and global spatial features. Bayesian Optimization and Genetic Algorithm are used to optimize hyperparameters for improving the performance of the model. The Hybrid Deep Neural Network (HDNN) is the last classifier with an accuracy of 97.4%. Based on performance accuracy, F1-score, the model is strong and reliable. We used Gradient-weighted Class Activation Mapping++ to explain how regions of interest in the brain influence our model's decisions. This study offers an interpretable and high-performing deep learning framework for the early and precise prediction of neurodegenerative disorders utilizing MRI imaging, thereby enhancing clinical decision-making and patient care.

A multi-kernel-based spatiotemporal modeling approach for energy transfer of complex thermal processes and its applications

Medical image segmentation plays a vital role in diagnostic imaging and treatment planning, especially for volumetric modalities such as MRI and CT scans. However, training high-performance deep learning models for 3D medical image segmentation requires large, annotated datasets, which are often siloed due to strict privacy laws like HIPAA and GDPR. Federated Learning (FL) offers a decentralized solution that enables collaborative training without sharing raw patient data, but its effectiveness is hindered by challenges such as data heterogeneity, communication overhead, and model degradation on non-IID datasets. In this study, we propose FedSegNet, a novel federated learning framework tailored for 3D medical image segmentation. FedSegNet integrates a Transformer-based U-Net architecture for capturing both local and global spatial features and introduces an Adaptive Aggregation Mechanism (AAM) to dynamically weigh client updates based on data quality, performance, and divergence. To reduce communication costs, the framework employs gradient sparsification and quantization techniques. We evaluate FedSegNet on multi-institutional datasets including BraTS, LiTS, and ACDC, using metrics such as Dice Similarity Coefficient and Hausdorff Distance. Results show that FedSegNet achieves up to 7.2% improvement in segmentation accuracy and 38% reduction in communication cost compared to existing methods, demonstrating its potential for secure, decentralized medical AI applications.

Periodic traffic prediction and analysis is essential for urbanisation and intelligent transportation systems (ITS). However, traffic prediction is challenging due to the nonlinear flow of traffic and its interdependencies on spatiotemporal features. Traffic flow has a long-term dependence on temporal features and a short-term dependence on local and global spatial features. It is strongly influenced by external factors such as weather and points of interest. Existing models consider long-term and short-term predictions in Euclidean space. In this paper, we design an attention-based encoder–decoder with stacked layers of LSTM to analyse multiscale spatiotemporal dependencies in non-Euclidean space to forecast traffic. The attention weights are obtained adaptively and external factors are fused with the output of the decoder to evaluate region-wide traffic predictions. Extensive experiments are conducted to evaluate the performance of the proposed attention-based non-Euclidean spatiotemporal network (ANST) on real-world datasets. The proposed model has improved prediction accuracy over previous methods. The insights obtained from traffic prediction would be beneficial for daily commutation and logistics.

EAAR: Efficient and Accurate Action Recognition model with enhanced spatio-temporal perception.

Vehicle re-identification aims to retrieve images of the same vehicle under different viewpoints, camera perspectives, and environmental conditions from a large-scale image database. However, significant challenges arise due to intra-class variations and inter-class similarities. Moreover, the difficulty in effectively utilizing multi-view information substantially limits further improvements in recognition accuracy. This paper proposes a novel multi-branch network named MGF-Net that enhances vehicle feature representation by jointly optimizing global and fine-grained feature extraction modules. In the global feature extraction module, a dual-dimensional channel–spatial adaptive feature re-weighting mechanism is implemented to amplify responses in vehicle-centered regions, suppress background interference, and improve the robustness of global features. In the fine-grained feature extraction module, a multi-stage feature interaction mechanism is constructed to capture detailed features and spatial semantic associations of key vehicle parts, such as the front grille and hood emblem, thereby avoiding the feature fragmentation issue caused by manual part partitioning. To further improve recognition performance in cross-view scenarios, a multi-view fusion strategy is introduced to dynamically aggregate feature representations from different viewpoints, thereby enhancing the model’s generalization ability. Additionally, both branches are optimized using classification loss and metric loss functions to refine the model from multiple perspectives. Extensive experiments on the challenging VeRi-776 and VehicleID datasets demonstrate that our approach achieves state-of-the-art performance and significantly improves vehicle re-identification accuracy.

ECO [20] is a newly proposed efficient video classification network that combines 2D and 3D networks. On this basis, we try to extract the global features composed of inter-frame relationships and fuse the global features with the initial frame feature, which can enhance the learning of video features to achieve better classification results. In this way, the new frame feature not only contains the information of itself but also contains the global feature which is related to other frames., so the module can more easily learn about distinguishing features. The main contribution of this paper is proposing a module for extracting global features to enhance video feature learning. It is very lightweight and basically does not add computation to the model. At the same time, it is flexible and easy to generalize to various network structures. Finally, based on the baseline the classification accuracy is increased by 0.8% on the HMDB51 dataset.

Pose-based action recognition has always been an important research field in computer vision. However, most existing pose-based methods are built upon human skeleton data, which cannot be used to exploit the feature of the motion-related object, i.e., a crucial clue of discriminating human actions. To address this issue, we propose a novel pose-flow relational model, which can benefit from both pose dynamics and optical flow. First, we introduce a pose estimation module to extract the skeleton data of the key person from the raw video. Second, a hierarchical pose-based network is proposed to effectively explore the rich spatial–temporal features of human skeleton positions. Third, we embed an inflated 3D network to capture the subtle cues of the motion-related object from optical flow. Additionally, we evaluate our model on four popular action recognition benchmarks (HMDB-51, JHMDB, sub-JHMDB, and SYSU 3D). Experimental results demonstrate that the proposed model outperforms the existing pose-based methods in human action recognition.