Spatio-Temporal Attention based Recurrent Neural Network for Next Location Prediction

In recent years, research into the mining of user check-in behavior for point-of-interest (POI) recommendations has attracted a lot of attention. Existing studies on this topic mainly treat such recommendations in a traditional manner—that is, they treat POIs as items and check-ins as ratings. However, users usually visit a place for reasons other than to simply say that they have visited. In this article, we propose an approach referred to as Urban POI-Walk (UPOI-Walk), which takes into account a user's social-triggered intentions (SI), preference-triggered intentions (PreI), and popularity-triggered intentions (PopI), to estimate the probability of a user checking-in to a POI. The core idea of UPOI-Walk involves building a HITS-based random walk on the normalized check-in network, thus supporting the prediction of POI properties related to each user's preferences. To achieve this goal, we define several user--POI graphs to capture the key properties of the check-in behavior motivated by user intentions. In our UPOI-Walk approach, we propose a new kind of random walk model—Dynamic HITS-based Random Walk—which comprehensively considers the relevance between POIs and users from different aspects. On the basis of similitude, we make an online recommendation as to the POI the user intends to visit. To the best of our knowledge, this is the first work on urban POI recommendations that considers user check-in behavior motivated by SI, PreI, and PopI in location-based social network data. Through comprehensive experimental evaluations on two real datasets, the proposed UPOI-Walk is shown to deliver excellent performance.

As the fundamental part of other Intelligent Transportation Systems (ITS) applications, short-term traffic volume prediction plays an important role in various intelligent transportation tasks, such as traffic management, traffic signal control and route planning. Although Neural-network-based traffic prediction methods can produce good results, most of the models can’t be explained in an intuitive way. In this paper, we not only proposed a model that increase the short-term prediction accuracy of the traffic volume, but also improved the interpretability of the model by analyzing the internal attention score learnt by the model. we propose a spatiotemporal attention mechanism-based multistep traffic volume prediction model (SAMM). Inside the model, an LSTM-based Encoder-Decoder network with a hybrid attention mechanism is introduced, which consists of spatial attention and temporal attention. In the first level, the local and global spatial attention mechanisms considering the micro traffic evolution and macro pattern similarity, respectively, are applied to capture and amplify the features from the highly correlated entrance stations. In the second level, a temporal attention mechanism is employed to amplify the features from the time steps captured as contributing more to the future exit volume. Considering the time-dependent characteristics and the continuity of the recent evolutionary traffic volume trend, the timestamp features and historical exit volume series of target stations are included as the external inputs. An experiment is conducted using data from the highway toll collection system of Guangdong Province, China. By extracting and analyzing the weights of the spatial and temporal attention layers, the contributions of the intermediate parameters are revealed and explained with knowledge acquired by historical statistics. The results show that the proposed model outperforms the state-of-the-art model by 29.51% in terms of MSE, 13.93% in terms of MAE, and 5.69% in terms of MAPE. The effectiveness of the Encoder-Decoder framework and the attention mechanism are also verified.

Accurate prediction of mesoscale eddy trajectories requires efficient models with large-size available data instances to capture the main eddy characteristics. However, there is a lack of salient attention mechanisms that can recognize the demand of extracting the aggregated features over multidimensional eddy data sources. Additionally, deep learning techniques are very important for eddy trajectory prediction to dynamically capture properties of mesoscale eddies in the South China Sea. In this article, we propose a spatio-temporal attention-based deep learning framework that can orchestrate heterogeneous data integration and propagation trajectory forecast together. It consists of a novel autoencoder equipped with channel and spatial attention mechanisms (CSA-encoder), and a gated recurrent unit (GRU) network with temporal attention layer (TA-GRU). CSA-encoder compresses stereoscopic eddy data with convolutional layers and generates the small-scale and high-quality dataset as the input of TA-GRU. The finer grained TA-GRU method is extended to accurately predict eddy trajectories with more valuable imagery information so that the temporal attention mechanism can automatically select relevant regions within the next 14 days. Our cross-validation results demonstrate that our framework averagely achieves a lower distance error (9 km) and 54% performance improvement over the baseline GRU technique in the next one day, and outperforms two state-of-the-art techniques of long short-term memory and recurrent neural network by 54.9% and 65.6%, respectively.

In recent years, researches on recommendation of urban Points-Of-Interest (POI), such as restaurants, based on social information have attracted a lot of attention. Although a number of social-based recommendation techniques have been proposed in the literature, most of their concepts are only based on the individual or friends' check-in behaviors. It leads to that the recommended POIs list is usually constrained within the users' or friends' living area. Furthermore, since context-aware and environmental information changes quickly, especially in urban areas, how to extract appropriate features from such kind of heterogeneous data to facilitate the recommendation is also a critical and challenging issue. In this paper, we propose a novel approach named Urban POI-Mine (UPOI-Mine) that integrates location-based social networks (LBSNs) for recommending users urban POIs based on the user preferences and location properties simultaneously. The core idea of UPOI-Mine is to build a regression-tree-based predictor in the normalized check-in space, so as to support the prediction of interestingness of POI related to each user's preference. Based on the LBSN data, we extract the features of places in terms of i) Social Factor, ii) Individual Preference, and iii) POI Popularity for model building. To our best knowledge, this is the first work on urban POI recommendation that considers social factor, individual preference and POI popularity in LBSN data, simultaneously. Through comprehensive experimental evaluations on a real dataset from Gowalla, the proposed UPOI-Mine is shown to deliver excellent performance.

We used ERPs to investigate whether temporal attention interacts with spatial attention and feature-based attention to enhance visual processing. We presented a visual search display containing one singleton stimulus among a set of homogenous distractors. Participants were asked to respond only to target singletons of a particular color and shape that were presented in an attended spatial position. We manipulated temporal attention by presenting a warning signal before each search display and varying the foreperiod (FP) between the warning signal and the search display in a blocked manner. We observed distinctive ERP effects of both spatial and temporal attention. The amplitudes for the N2pc, SPCN, and P3 were enhanced by spatial attention indicating a processing benefit of relevant stimulus features at the attended side. Temporal attention accelerated stimulus processing; this was indexed by an earlier onset of the N2pc component and a reduction in reaction times to targets. Most importantly, temporal attention did not interact with spatial attention or stimulus features to influence visual processing. Taken together, the results suggest that temporal attention fosters visual perceptual processing in a visual search task independently from spatial attention and feature-based attention; this provides support for the nonspecific enhancement hypothesis of temporal attention.

Point of Interest (POI) recommendation is to use the user check-in data in location-based Social Networks (LBSN) to predict the next Location that the user will visit. With the rapid development of LBSN, users' check-in information (rating information, geographic location, check-in sequence, social network, etc.) has become more and more accessible. How to effectively fuse and model these multi-source heterogeneous data is crucial to personalized POI recommendations. This paper presents a POI recommendation model based on improved factorization machine and Bert. By integrating social influence on the basis of factorization machine, Bert was used to extract the check-in sequence features. Experiments on real data sets show that our model performs better than the traditional POI recommended model.

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

NRDL: Decentralized user preference learning for privacy-preserving next POI recommendation

A Contextual Recurrent Collaborative Filtering framework for modelling sequences of venue checkins

With the thriving of location-based social networks, a large number of user check-in data have been accumulated. Tasks such as the prediction of the next check-in location can be addressed through the usage of LBSN data. Previous work mainly uses the historical trajectories of users to analyze users' check-in behavior, while the social information of users was rarely used. In this paper, we propose a unified location prediction framework to integrate the effect of history check-in and the influence of social circles. We first employ the most frequent check-in model (MFC) and the user-based collaborative filtering model (UCF) to capture users' historical trajectories and users' implicit preference, respectively. Then we use the multi-social circle model (MSC) to model the influence of three social circles. Finally, we evaluate our location prediction framework in the real-world data sets, and the experimental results show that our model performs better than the state-of-the-art approaches in predicting the next check-in location.

With the increasing popularity of location-aware Internet-of-Vehicle services, the next-Point-of-Interest (POI) recommendation has gained significant research interest, predicting where drivers will go next from their sequential movements. Many researchers have focused on this problem and proposed solutions. Machine learning-based methods (matrix factorization, Markov chain, and factorizing personalized Markov chain) focus on a POI sequential transition. However, they do not recommend the user’s position for the next few hours. Neural network-based methods can model user mobility behavior by learning the representations of the sequence data in the high-dimensional space. However, they just consider the influence from the spatiotemporal dimension and ignore many important influences, such as duration time at a POI (Point of Interest) and the semantic tags of the POIs. In this paper, we propose a novel method called multi-dimension spatial–temporal recurrent neural networks (MST-RNN), which extends the ST-RNN and exploits the duration time dimension and semantic tag dimension of POIs in each layer of neural networks. Experiments on real-world vehicle movement data show that the proposed MST-RNN is effective and clearly outperforms the state-of-the-art methods.

With the development of Internet of Things technology, the will to make cities smarter is growing. In the context of smart cities, where people are surrounded by a tremendous number of points of interests (POIs), POI recommendation is of great significance. The massive amount of user check-in data collected by location-based social networks (LBSNs) can help users explore new places via POI recommendations. With rich auxiliary data becomeing available in LBSNs, purely exploiting users' check-in information for POI recommendation is not sufficient. Although several attempts have been done to employ geographical influence to enhance POI recommendation, they simply use a certain distribution function to measure the geographical influence between POIs, which may lead to biased results. To this end, this article proposes a geography-aware inductive matrix completion (GAIMC) approach for personalized POI recommendation. Specifically, the GAIMC model consists of two parts, including geographic feature extraction via a Gaussian mixture model (GMM) and inductive matrix completion for recommendation. The GAIMC model first captures the geographical influence among users and POIs by the GMM, which can mine clustering information with hierarchical structures based on the users' check-in data and POIs' location information. Then, a matrix-completion method called inductive matrix completion, which can incorporate geographical features with the user-POI association metric, is utilized to recommend POIs. The experimental results on two real-world LBSN data sets demonstrate that our proposed model can achieve the best recommendation performance in comparison with the state-of-the-art counterparts.

Recommending a limited number of Point-of-Interests (POIs) a user will visit next has become increasingly important to both users and POI holders for Location-Based Social Networks (LBSNs). However, POI recommendation is a challenging task since complex sequential patterns and rich contexts are contained in extremely sparse user check-in data. Recent studies show that embedding techniques effectively incorporate POI contextual information to alleviate the data sparsity issue, and Recurrent Neural Network (RNN) has been successfully employed for sequential prediction. Nevertheless, existing POI recommendation approaches are still limited in capturing user personalized preference due to separate embedding learning or network modeling. To this end, we propose a novel unified spatio-temporal neural network framework, named PPR, which leverages users’ check-in records and social ties to recommend personalized POIs for querying users by joint embedding and sequential modeling. Specifically, PPR first learns user and POI representations by joint modeling User-POI relation, sequential patterns, geographical influence, and social ties in a heterogeneous graph and then models user personalized sequential patterns using the designed spatio-temporal neural network based on LSTM model for the personalized POI recommendation. Furthermore, we extend PPR to an end-to-end recommendation model by jointly learning node representations and modeling user personalized sequential preference. Extensive experiments on three real-world datasets demonstrate that our model significantly outperforms state-of-the-art baselines for successive POI recommendation in terms of Accuracy, Precision, Recall and NDCG. The source code is available at: https://www.anonymous.4open.science/r/DSE-1BEC.

PSTFormer: A novel parallel spatial-temporal transformer for remaining useful life prediction of aeroengine

Spatial correlation and temporal attention-based LSTM for remaining useful life prediction of turbofan engine