Grounding language models in spatiotemporal context

There is context-related temporal and spatial information in video. How to make better use of these information is the key to improving the accuracy and speed of video object detection. The method of processing spatial and temporal information through 3D convolution is better than that of processing spatial information first and then processing time dimension information. There are two reasons for this assertion. One is that the 3D convolution can process spatial information and temporal information at the same time which will not divide them into two parts in feature extraction. The other is that the human visual derivation mechanism needs to memorize multi frames and process a video as a whole. After the training test on NFS dataset, it is verified that 3D convolution has stronger ability to extract temporal and spatial information. In addition, this paper also proposes a downward sampling method by connecting the low-resolution layer and the high-resolution layer, which can effectively improve the FPN model and enrich its high-level feature representation. After training and testing with different number of downward sample modules on MOT16 dataset, the improvement of model performance by down-sampling module is verified. Finally, this paper combines the downward sample module with the 3D network, and obtains the best test results on the NFS dataset.

In the context of automotive driving assistance, this paper describes a generic (i.e. applicable to both vehicle interior and exterior scenes) vision based approach for scene content analysis. It makes use of temporal and spatial information from a stereoscopic sequence of images to localize objects and estimate their position and motion. The proposed method is divided into three steps. First, image features are selected, tracked and reconstructed in the 3D world space. Second, a clustering step is processed in the 5D space made of the positions and 2D motions parameters. The last step is devoted to clusters interpretation: it is out of the scope of the paper, however orientations are given to illustrate the capabilities of the proposed approach. The paper is organized as follows: first, the use of temporal and spatial information from a stereoscopic sequence is investigated. A state of the art of existing methods is presented. Then, a generic approach for object segmentation is proposed. Lastly, experimental results are presented.

Modern industrial plants generally consist of multiple manufacturing units, and the local correlation within each unit can be used to effectively alleviate the effect of spurious correlation and meticulously reflect the operation status of the process system. Therefore, the local correlation, which is called spatial information here, should also be taken into consideration when developing the monitoring model. In this study, a cascaded monitoring network (MoniNet) method is proposed to develop the monitoring model with concurrent analytics of temporal and spatial information. By implementing convolutional operation to each variable, the temporal information that reveals dynamic correlation of process data and spatial information that reflects local characteristics within individual operation unit can be extracted simultaneously. For each convolutional feature, a submodel is developed and then all the submodels are integrated to generate a final monitoring model. Based on the developed model, the operation status of the newly collected sample can be identified by comparing the calculated statistics with their corresponding control limits. Similar to the convolutional neural network (CNN), the MoniNet can also expand its receptive field and capture deeper information by adding more convolutional layers. Besides, the filter selection and submodel development in MoniNet can be replaced to generalize the proposed network to many existing monitoring strategies. The performance of the proposed method is validated using two real industrial processes. The illustration results show that the proposed method can effectively detect process anomalies by concurrent analytics of temporal and spatial information.

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

Episodic memory involves the processing of spatial and temporal aspects of personal experiences. The lateral entorhinal cortex (LEC) plays an essential role in subserving memory. However, the mechanisms by which LEC integrates spatial and temporal information remain elusive. Here, we recorded LEC neurons while male rats performed one-dimensional tasks. Many LEC cells displayed spatial firing fields and demonstrated selectivity for traveling directions. Furthermore, some LEC neurons changed the firing rates of their spatial rate maps during a session (rate remapping). Importantly, this temporal modulation was consistent across sessions, even when the spatial environment was altered. Notably, the strength of temporal modulation was greater in LEC compared to other brain regions, such as the medial entorhinal cortex, CA1, and CA3. Thus, we demonstrate spatial rate mapping in LEC neurons, which may serve as a coding mechanism for temporal context, and allow for flexible multiplexing of spatial and temporal information.

Home Network-Attached Storage (NAS) provides an easy way for data sharing and backup among multiple consumer electronic devices in home networks. Because of large capacity and cost effectiveness, disks are widely adopted in home NAS devices. In addition, data writes are common for many home NAS devices since these devices are usually used for data storage and backup. A writeback policy selects which dirty buffers are to be flushed to the disk, which is critical to the system performance under write-intensive workloads. In this paper, an intelligent writeback policy for home NAS called TESA is proposed. In contrast to most existing writeback policies, TESA considers both temporal and spatial information of the dirty buffers to improve the writeback performance. Considering the temporal information helps to reduce the frequency of writing back recently used dirty buffers, and hence leads to reduced write traffic. Considering the spatial information causes a reduction in the overall seek time and rotation delay of the dirty buffer writeback. The TESA writeback policy was implemented on a NAS evaluation board running Linux kernel. The performance results shows that TESA yields a significant performance improvement (i.e., up to 33.1%) over the original writeback policy of Linux. Moreover, TESA outperforms the policy that considers only the temporal information of the dirty buffers by up to 21.0%, and it also has up to 10.9% performance improvement over a previous policy that considers both the temporal and spatial information.

The Acquisition and Long-Term Retention of Temporal, Spatial, and Item Information

In addition to the strong correlation in the internal pixels of the image, the pixels in the two phases of image have a certain correlation, that is, temporal context information. Conditional Random Field (CRF) models not only model spatial context information, but also fuse temporal context information. A spatio-temporal model based on temporal context information and spatial context information is proposed to improve the classification accuracy of remote sensing images. Use of two-phase data, the temporal context information between two phases and the spatial context information between pixels within a single temporal phase are considered, and High-order CRF model the spatio-temporal context information. Make full use of spatio-temporal context information of two-phase image to improve the classification accuracy. The experimental results show that the proposed method has better classification accuracy than the classification accuracy without temporal context information.

With the rapid development of mobile computing and Web services, a huge amount of data with spatial and temporal information have been collected everyday by smart mobile terminals, in which an object is described by its spatial information and temporal information. Motivated by the significance of spatio-temporal range search and the lack of efficient search algorithm, in this paper, we study the problem of spatio-temporal range search (STRS), a novel index structure is proposed, called HOC-Tree, which is based on Hilbert curve and OC-Tree, and takes both spatial and temporal information into consideration. Based on HOC-Tree, we develop an efficient algorithm to solve the problem of spatio-temporal range search. Comprehensive experiments on real and synthetic data demonstrate that our method is more efficient than the state-of-the-art technique.

Spatial and temporal saliency based four-stream network with multi-task learning for action recognition

The analysis of a multitemporal sequence of images of a given site makes it possible to exploit temporal information in addition to spectral and spatial information, and therefore represents a way to improve the accuracy with respect to the non-contextual single-time classification. The proposed contextual multitemporal classification scheme consists of two stages of multilayer perceptron (MLP) neural networks for each single-time image of the multitemporal sequence. The first stage is a one-hidden layer MLP whose ro/spl circ/le is to estimate, for each pixel, the single-time posterior probability of each class, given the feature vector. These probability estimates represent spectral information; in addition, they are utilized to generate a non-contextual classification map. The neighboring class labels of a given pixel in the non-contextual classification map are exploited to extract spatial information, while temporal information is deduced from the non-contextual maps produced by the remaining single-time images in the multitemporal sequence. Spatial and temporal contextual information together with spectral information serve as inputs for the second stage network of the classification scheme where the fusion takes place. As the network configuration can influence the classification performances, three MLP-based configurations are investigated. Experimental results on a multitemporal data set consisting of two multisensor (Landsat TM and ERS-1 SAR) images are presented. The performances of the proposed methods are discussed and compared with those obtained by a reference classifier based on the Markov random fields fusion approach in terms of classification accuracy. The results show that the proposed fusion approach based on neural networks may represent an interesting solution to the problem of multitemporal contextual fusion.

This paper proposes a novel algorithm for automatically and quickly extracting the external edge of the moving object. The algorithm uses both spatial and temporal information, which is abundant in image sequences and has not usually been efficiently used in conventional algorithm. Especially most attention of this algorithm is paid for the moving object external-edges. This orientation is designed to deal with the overlap of moving objects in successive frames. After the temporal segmentation the external edge of moving object is detected. Differing from current algorithms the fusing way of the temporal and spatial information has been progressed. Here the algorithm only uses spatial information of the temporal external edge to refine the edge points for getting the accurate boundary. This method will greatly decrease the compute and achieve the fast process in the real-time system. Though the simulation, this algorithm can get perfect result.

Location prediction has attracted a significant amount of research effort. Given an object's recent movements and a future time, the goal of location prediction is to predict the location of this object at the future time specified. Prior works have elaborated on mining association relationships among regions, in which objects frequently appear, to predict locations. Association relationships among regions are represented as association rules. By exploring association relationships among regions, prior works are able to have a good accuracy for location prediction. However, with a large amount of trajectory data produced, a huge amount of association rules is expected. Furthermore, trajectory data has both the spatial and temporal information. To further enhance the accuracy of location prediction, one could utilize not only spatial information but also temporal information to estimate locations of objects. In this paper, we propose a spatial-temporal trajectory model (abbreviated as STT) to capture movement behaviors of objects. STT is represented as a probabilistic suffix tree with both spatial and temporal information of movements. Note that STT is able to discover sequential traversal relationships among regions and, for each region, STT derives the corresponding probabilities about the time when objects appear. With the nature of probabilistic suffix tree, we could use a compact structure to capture movement behavior of objects compared to association rules proposed. In light of STT, we further propose an algorithm to traverse STT for location prediction. By exploring both the spatial and temporal information of STT, the accuracy of location prediction is improved. To evaluate our proposed STT and prediction algorithm, we conduct experiments on the synthetic dataset generated from real datasets. Experimental results shows that our proposed STT is able to capture both spatial and temporal patterns of movement behaviors and, by exploring STT, our proposed prediction algorithm outperforms existing works.

The present work explores an interactive model of spatial and temporal information in map memory. In four experiments, participants learned a map with temporal and spatial information confounded or unconfounded. Attentional and representational levels of information were made apparent through tasks that tap spatial, temporal, or other information. Learning criteria emphasizing sequential order or location imposed differential weighting of the information types in memory. Results indicate that map memory is spatial, but also interacts with the order in which map locations are encountered. Findings show flexibility in allocating attention and information indexing of location and sequential order information in map learning.

This paper deals with a video content classification based on spatial and temporal information. The main goal of the paper is to find out whether the non-expert test subjects are able to correctly recognize the spatial and temporal information and how precise is the corresponding parameterization of video content based on these two attributes. Moreover, how well do the objective values provided by the tool deployed in this experiment correlate with the subjective ones. The subjective measurement has shown that the non-expert test subjects are able to correctly recognize the spatial and temporal information only for the extreme scenes in terms of the spatial and temporal information but have experienced some problems when it comes to a more granular classification. When it comes to the objective assessment, the SI and TI values provided by the tool used in this experiment correlate quite well with the subjective data obtained from the basic test for the spatial information.