Patch-Wise Spatial-Temporal Quality Enhancement for HEVC Compressed Video.

A network traffic prediction model based on reinforced staged feature interaction and fusion

Graph convolutional network – Long short term memory neural network- multi layer perceptron- Gaussian progress regression model: A new deep learning model for predicting ozone concertation

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

Gait recognition provides a more convenient way for human identification, as it can identify person with less cooperation and intrusion compared to other biometric features. Current gait recognition frameworks utilize a template to extract temporal feature or regard the whole person as a unit, and they obtain limited temporal information and fine-grained features. To overcome this problem, we propose a network consisting of two parts: Temporal Feature Fusion (TFF) and Fine-grained Feature Extraction (FFE). First, we extract the most representative temporal information from raw gait sequences by TFF. Next, we use the idea of partial features on fused temporal features to extract more fine-grained spatial block features. It is worth mentioning that the proposed algorithm provides an effective feature extraction framework for complex gait recognition, as it focuses on the temporal fusion for representative information, and the extraction of the fine-grained spatial features. Extensive experiments illustrated that we have an outstanding performance on CASIA-B and mini-OUMVLP compared to other state-of-the-art methods including GaitSet and GaitNet. In particularly, the average rank-1 accuracy of all probe views on normal walking condition (NM) achieve 95.7%.

Various techniques using artificial intelligence (AI) have resulted in a significant contribution to field of medical image and video-based diagnoses, such as radiology, pathology, and endoscopy, including the classification of gastrointestinal (GI) diseases. Most previous studies on the classification of GI diseases use only spatial features, which demonstrate low performance in the classification of multiple GI diseases. Although there are a few previous studies using temporal features based on a three-dimensional convolutional neural network, only a specific part of the GI tract was involved with the limited number of classes. To overcome these problems, we propose a comprehensive AI-based framework for the classification of multiple GI diseases by using endoscopic videos, which can simultaneously extract both spatial and temporal features to achieve better classification performance. Two different residual networks and a long short-term memory model are integrated in a cascaded mode to extract spatial and temporal features, respectively. Experiments were conducted on a combined dataset consisting of one of the largest endoscopic videos with 52,471 frames. The results demonstrate the effectiveness of the proposed classification framework for multi-GI diseases. The experimental results of the proposed model (97.057% area under the curve) demonstrate superior performance over the state-of-the-art methods and indicate its potential for clinical applications.

Objectives: Brain arteriovenous malformation (AVM) is one of the most common causes of intracranial hemorrhage in young adults, and its expeditious diagnosis on digital subtraction angiography (DSA) is essential for clinical decision-making. This paper firstly proposed a deep learning network to extract vascular time-domain features from DSA videos. Then, the temporal features were combined with spatial radiomics features to build an AVM-assisted diagnosis model.Materials and method: Anteroposterior position (AP) DSA videos from 305 patients, 153 normal and 152 with AVM, were analyzed. A deep learning network based on Faster-RCNN was proposed to track important vascular features in DSA. Then the appearance order of important vascular structures was quantified as the temporal features. The structure distribution and morphological features of vessels were quantified as 1,750 radiomics features. Temporal features and radiomics features were fused in a classifier based on sparse representation and support vector machine. An AVM diagnosis and grading system that combined the temporal and spatial radiomics features of DSA was finally proposed. Accuracy (ACC), sensitivity (SENS), specificity (SPEC), and area under the receiver operating characteristic curve (AUC) were calculated to evaluate the performance of the radiomics model.Results: For cerebrovascular structure detection, the average precision (AP) was 0.922, 0.991, 0.769, 0.899, and 0.929 for internal carotid artery, Willis circle, vessels, large veins, and venous sinuses, respectively. The mean average precision (mAP) of five time phases was 0.902. For AVM diagnosis, the models based on temporal features, radiomics features, and combined features achieved AUC of 0.916, 0.918, and 0.942, respectively. In the AVM grading task, the proposed combined model also achieved AUC of 0.871 in the independent testing set.Conclusion: DSA videos provide rich temporal and spatial distribution characteristics of cerebral blood vessels. Clinicians often interpret these features based on subjective experience. This paper proposes a scheme based on deep learning and traditional machine learning, which effectively integrates the complex spatiotemporal features in DSA, and verifies the value of this scheme in the diagnosis of AVM.

Timely and accurate traffic flow information is essential for the realization of intelligent transportation systems (ITS). For the existing traffic flow prediction models, only the temporal features were extracted but the spatial features were neglects. A short-term traffic flow prediction model based on Pearson Correlation Coefficient (PCC) and Bi-directional Long Short-Term Memory Network (BiLSTM) model was proposed. The PCC was used to mine the spatial dependence of the road network, and the temporal features of traffic flow were mined using BiLSTM model. The extracted temporal and spatial features were combined to achieve short-term flow prediction. The experimental results show that the PCC-BiLSTM model has higher prediction accuracy and lower error than the prediction models considering only temporal features, which verifies the effectiveness of traffic flow prediction in consideration of both temporal and spatial characteristics.

Multi-view fusion neural network for traffic demand prediction

As ride-hailing services become increasingly popular, being able to accurately predict demand for such services can help operators efficiently allocate drivers to customers, and reduce idle time, improve traffic congestion, and enhance the passenger experience. This paper proposes UberNet, a deep learning convolutional neural network for short-time prediction of demand for ride-hailing services. Exploiting traditional time series approaches for this problem is challenging due to strong surges and declines in pickups, as well as spatial concentrations of demand. This leads to pickup patterns that are unevenly distributed over time and space. UberNet employs a multivariate framework that utilises a number of temporal and spatial features that have been found in the literature to explain demand for ride-hailing services. Specifically, the proposed model includes two sub-networks that aim to encode the source series of various features and decode the predicting series, respectively. To assess the performance and effectiveness of UberNet, we use 9 months of Uber pickup data in 2014 and 28 spatial and temporal features from New York City. We use a number of features suggested by the transport operations and travel behaviour research areas as being relevant to passenger demand prediction, e.g., weather, temporal factors, socioeconomic and demographics characteristics, as well as travel-to-work, built environment and social factors such as crime level, within a multivariate framework, that leads to operational and policy insights for multiple communities: the ride-hailing operator, passengers, third-part location-based service providers and revenue opportunities to drivers, and transport operators such as road traffic authorities, and public transport agencies. By comparing the performance of UberNet with several other approaches, we show that the prediction quality of the model is highly competitive. Further, Ubernet’s prediction performance is better when using economic, social and built environment features. This suggests that Ubernet is more naturally suited to including complex motivators of travel behavior in making real-time demand predictions for ride-hailing services.

Face expression recognition in videos is a challenging task as it involves extracting and combining high level temporal and spatial features from the video sequences to automatically identify human expression categories. In this paper, we propose a Hybrid feature extractor, analyze various deep learning classifiers and their performance. Addressing illumination invariance and scale invariance, this method initially pre-processes the input video to remove noise and uses Viola jones algorithm to detect face region; then the high significant features, such as temporal features, spatial domain features, and the frequency domain features are extracted. Further, dimensionality issues are handled effectively using the Principle Component Analysis (PCA). Finally, the fused features are fed to train CNN-LSTM, a hybrid deep learning classifier for identifying face expressions. Experiment was conducted on SAVEE, RAVDESS and AFEW Datasets. The proposed method obtained an overall accuracy, precision, and recall of 90.5%, 92.39%, 88.18%, respectively.KeywordsFace expression recognitionViola jones algorithmPrinciple component analysisLocal binary patternAnd hybrid deep learning classifier

Evaluating cognitive load using electroencephalogram (EEG) signals is a crucial research area in the field of Brain-Computer Interfaces (BCI). However, achieving high accuracy and generalization in feature extraction and classification for cognitive load assessment remains a challenge, primarily due to the low signal-to-noise ratio of EEG signals and the inter-individual variability in EEG data. In this study, we propose a novel deep learning architecture that integrates temporal information features and functional connectivity features to enhance EEG signal analysis. Functional connectivity features capture inter-channel information, while temporal features are extracted from continuous signal segments using a Long Short-Term Memory (LSTM) network enhanced with an attention mechanism. The fusion strategy combines these two information streams to leverage their complementary strengths, resulting in improved classification performance. We evaluated our architecture on two publicly available datasets, and the results demonstrate its robustness in cognitive load recognition. Achieving performance comparable to the best existing methods on two public datasets. Ablation studies further substantiate the contributions of each module, demonstrating the importance of combining temporal and functional connectivity features for optimal results. These findings underscore the robustness and versatility of the proposed approach, paving the way for more effective EEG-based BCI applications.

We propose a parallel network with spatial–temporal attention for video-based person re-identification. Many previous video-based person re-identification methods use two-dimensional convolutional neural networks to extract spatial features, then, temporal features are extracted by temporal pooling or recurrent neural networks. Unfortunately, these series networks will cause the loss of spatial information when extracting temporal information. Different from previous methods, our parallel network can extract temporal and spatial features simultaneously, which can effectively reduce the loss of space information. In addition, we design a global temporal attention module, which obtains the attention weight through the correlation between the current frame and all the frames in the sequence. At the same time, the temporal module can act on the information extraction of spatial module. In this way, we can increase the temporal and spatial constraints. Experiments show that our method can effectively improve the re-id accuracy, better than the state-of-the-art methods.

In complex tracking scenarios of multitarget occlusion, especially when the target is seriously occluded, feature loss can lead to tracking failure. In this article, we develop a multitarget occlusion tracking algorithm based on a 3-D spatio-temporal context graph model, which can fully utilize the targets’ RGB-D information. Specifically, we first extract target’s 3-D features containing appearance, motion, spatial, and temporal features. Then, the spa’Io-temporal context features between different targets are also be acquired. Furthermore, the attribute relational graph (ARG) model with target’s 3-D spatio-temporal context features is constructed, which combines spatial features in the current frame and temporal features between consecutive frames. Moreover, if the target is determined to be occluded according to the matching results in the current frame, the probability relaxation method is used to assign the optimal tracking label to clustering blocks in occlusion area by matching these ARG models, which can address the multitarget cross occlusion problem. Extensive experiments on the Princeton Tracking Benchmark Datasets clearly demonstrate that our method can achieve superior performance against the typical algorithms proposed in recent years.

Deep learning based fault diagnosis methods have been widely studied for rolling bearings. However, for the existing such methods, the spatial and temporal features cannot be fully extracted and not considered. In order to solve the problems, a multi-attention parallel convolutional neural network-gate recurrent unit (MAPCG) fault diagnosis model is proposed in this paper. Firstly, variational modal decomposition(VMD) is used to obtain the eigenvectors through the original data of rolling bearing, which is utilized as the input of the proposed model. CNN and GRU are used to extract the spatial and temporal features as two parallel branches respectively. In addition, each branch is embedded with a specific attention module to distinguish the importance of features. Finally, the spatial feature and the temporal features are combined for the fault diagnosis. A mechanical fault simulation experiment platform is built for the experimental verification of the proposed model. And the experimental results indicates that the proposed model has superior fault diagnosis accuracy compared with existing models.

Lane detection plays an important role in autonomous driving. For video instance lane detection, both global spatial and temporal information is significantly important. However, the global spatial features and the temporal features are not been well exploited in recent studies. In this work, we address the video instance lane detection task by capturing global context based on non-local attention network. Specifically, we designed a twin non-local attention network to extract long-range dependencies along the spatial and temporal dimensions, respectively. Meanwhile, the global spatial and temporal features can be adaptively fused by gating mechanisms for better results. In addition, the frame-similarity loss is proposed to further exploit the information of adjacent frames. The experimental results on the video instance lane detection (VIL-100) dataset verify that our method achieves better results compared with other comparison methods. Ablation experiments further demonstrate the effectiveness of each sub-module.