PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction

Non-probability sampling network based on anomaly pedestrian trajectory discrimination for pedestrian trajectory prediction

Pedestrian trajectory prediction is fundamental to a wide range of scientific research work and industrial applications. Most of the current advanced trajectory prediction methods incorporate context information such as pedestrian neighbourhood, labelled static obstacles, and the background scene into the trajectory prediction process. In contrast to these methods which require rich contexts, the method in our paper focuses on predicting a pedestrian's future trajectory using his/her observed part of the trajectory only. Our method, which we refer to as LVTA, is a Location-Velocity-Temporal Attention LSTM model where two temporal attention mechanisms are applied to the hidden state vectors from the location and velocity LSTM layers. In addition, a location-velocity attention layer embedded inside a tweak module is used to improve the predicted location and velocity coordinates before they are passed to the next time step. Extensive experiments conducted on three large benchmark datasets and comparison with eleven existing trajectory prediction methods demonstrate that LVTA achieves competitive prediction performance. Specifically, LVTA attains 9.19 pixels Average Displacement Error (ADE) and 17.28 pixels Final Displacement Error (FDE) for the Central Station dataset, and 0.46 metres ADE and 0.92 metres FDE for the ETH&UCY datasets. Furthermore, evaluation on using LVTA to generate trajectories of different prediction lengths and on new scenes without the need of retraining confirms that it has good generalizability.

Pedestrian trajectory prediction via physical-guided position association learning

Pedestrian trajectory prediction is a critical research area with numerous domains, e.g., blind navigation, autonomous driving systems, and service robots. There exist two challenges in this research field: spatio-temporal interaction modeling among pedestrians and the uncertainty of pedestrian trajectories. To tackle these challenges, we propose a spatio-temporal interaction aware and trajectory distribution aware graph convolution network. First, we propose a spatio-temporal interaction aware module that integrates a graph convolutional network and self-attention mechanism to model spatio-temporal interactions among pedestrians. Second, we design a trajectory distribution aware module to learn latent trajectory distribution information from the measured trajectories at observed and future times. This can provide knowledge-rich trajectory distribution information for the multimodality of the predicted trajectories. Finally, to address the problem of the propagation and accumulation of prediction errors, we design a trajectory decoder to generate the multimodal future trajectories. The proposed model is evaluated utilizing videos recorded by a camera sensor in crowded areas and can be applied to predict multiple pedestrians’ future trajectories from in-vehicle cameras. Experimental results demonstrate that the proposed approach can achieve superior results on the average displacement error (ADE) and final displacement error (FDE) metrics to state-of-the-art approaches and can predict socially acceptable future trajectories.

Understanding crowd motion dynamics and forecasting the future pedestrian trajectories are critical to various applications, e.g. autonomous driving and surveillance system. This task is challenging because when pedestrians plan the future paths in real crowd scenes, they will distinguish the priorities of following their predetermined destinations and responding to the motion behaviors of neighboring pedestrians. However, most of the existing methods ignore the problem of intention-interaction trade-off. In this paper, we tackle this problem by a hierarchical network, which achieves dynamically reasoning predetermined destinations and future trajectories. A novel graph structure called Intention-Interaction Graph (IIG) is designed to jointly model the self intentions and social interactions. To aggregate information in IIG, Interaction Gated Graph Attention Networks (IGGAN) consisting of a gate mechanism and an attention mechanism is proposed, thus achieving reasoning the influence degree of neighboring pedestrians and destinations. Experimental results on multiple widely used pedestrian trajectory prediction datasets, including two datasets in ETH and three datasets in UCY, demonstrate the effectiveness of the proposed model.

Pedestrian path prediction is a very challenging problem because scenes are often crowded or contain obstacles. Existing state-of-the-art long short-term memory (LSTM)-based prediction methods have been mainly focused on analyzing the influence of other people in the neighborhood of each pedestrian while neglecting the role of potential destinations in determining a walking path. In this article, we propose classifying pedestrian trajectories into a number of route classes (RCs) and using them to describe the pedestrian movement patterns. Based on the RCs obtained from trajectory clustering, our algorithm, which we name the prediction of pedestrian paths by LSTM (PoPPL), predicts the destination regions through a bidirectional LSTM classification network in the first stage and then generates trajectories corresponding to the predicted destination regions through one of the three proposed LSTM-based architectures in the second stage. Our algorithm also outputs probabilities of multiple predicted trajectories that head toward the destination regions. We have evaluated PoPPL against other state-of-the-art methods on two public data sets. The results show that our algorithm outperforms other methods and incorporating potential destination prediction improves the trajectory prediction accuracy.

Physics constrained pedestrian trajectory prediction with probability quantification

This paper develops a pedestrian potentially dangerous behaviour prediction method based on attention‐long‐short‐term memory (Attention‐LSTM) architecture to predict pedestrian trajectory and intention for the unexpected pedestrian crossing accident avoidance. To extract the road scene information for short periods of time, and improve the accuracy of subsequent intention inference and trajectory prediction, the panoramic segmentation is used to extrapolate pedestrian instances and segment areas of the environment. Next, an encoder–decoder framework based on Attention‐LSTM model is proposed to infer a pedestrian's intention to run or walk out into oncoming traffic straight and to predict the future trajectory. The proposed network involves two parts: temporal feature encoder and multi‐task decoder. The temporal feature encoder is mainly used to selectively emphasize the temporal features using attention mechanism, and then LSTM is employed for its encoding. In the multi‐task decoder, a multi‐head self‐attention mechanism and LSTM are used to forecast the pedestrians’ intention and future trajectory, respectively. Extensive experiments on pedestrian intention estimation (PIE) datasets demonstrate that the authors’ proposed approach surpasses prior studies in terms of prediction accuracy in trajectory and intention. This study can not only effectively avoid serious accidents caused by illegal road crossing but also achieve early warning for collision avoidance.

Forecasting pedestrian trajectories in dynamic scenes remains a critical problem in various applications, such as autonomous driving and socially aware robots. Such forecasting is challenging due to human-human and human-object interactions and future uncertainties caused by human randomness. Generative model-based methods handle future uncertainties by sampling a latent variable. However, few studies explored the generation of the latent variable. In this work, we propose the trajectory predictor with pseudo Oracle (TPPO), which is a generative model-based trajectory predictor. The first pseudo oracle is pedestrians’ moving directions, and the second one is the latent variable estimated from ground truth trajectories. A social attention module is used to aggregate neighbors’ interactions based on the correlation between pedestrians’ moving directions and future trajectories. This correlation is inspired by the fact that pedestrians’ future trajectories are often influenced by pedestrians in front. A latent variable predictor is proposed to estimate latent variable distributions from observed and ground-truth trajectories. Moreover, the gap between these two distributions is minimized during training. Therefore, the latent variable predictor can estimate the latent variable from observed trajectories to approximate that estimated from ground-truth trajectories. We compare the performance of TPPO with related methods on several public datasets. Results demonstrate that TPPO outperforms state-of-the-art methods with low average and final displacement errors. The ablation study shows that the prediction performance will not dramatically decrease as sampling times decline during tests.

Pedestrian trajectory prediction is imperative in specific fields, such as crowd management and collision prevention in automated driving environments. In this study, a novel long-short-term memory (LSTM)-based deep neural network capable of simulating the different walking behaviours of individuals with and without disabilities was designed. This network consists of three modules: the Disability module, the Environmental module, and the Trajectory Prediction module. Data from a large-scale pedestrian walking behaviour experiment involving individuals with disabilities were used to train and test the network. These data correspond to several experiments. Each experiment attempts to capture the essence of individuals’ walking behaviour in different situations. By sequencing and normalising the input data and applying regularisation techniques, the network was successfully trained. The results were compared to state-of-the-art models, demonstrating that the network can predict pedestrians’ trajectories more accurately, especially when pedestrian heterogeneity is involved.

Pedestrian trajectory prediction can provide decision support for self-driving cars and robots to avoid potential collision risks. Modelling of social relationships and multimodal prediction are core challenges in pedestrian trajectory prediction. However, there are two problems with existing methods: (1) these methods usually focus on modelling pairwise or group interactions, ignoring the importance of high-order interactions; and (2) existing methods struggle to balance the performance between deterministic and multimodal predictions. To address these issues, we propose a higher-order graph attention network for pedestrian trajectory prediction. Specifically, we first construct spatial and temporal graphs to model social interactions and temporal dependencies. To achieve high-order interaction modelling, we design a higher-order graph attention module to capture pedestrian social and temporal interactions of different orders by stacking multiple layers of graph attention. Subsequently, we design a deformable spatial-temporal sparse module to adaptively eliminate social and temporal redundant interactions. To balance deterministic and multimodal prediction performance, we design a two-stage prediction strategy. The first stage predicts pedestrian deterministic trajectories, and the second stage refines pedestrian deterministic trajectories to complete both deterministic and multimodal predictions. Quantitative analysis on the ETH, UCY, and SDD datasets demonstrates that our method outperforms existing methods in both deterministic and multimodal predictions. Qualitative analysis shows that our method can effectively describe complex social relationships and accomplish reliable trajectory prediction.

Pedestrian behavior prediction is essential to enable safe and efficient driving of intelligent vehicles on urban traffic environment. This article presents a novel framework for pedestrian trajectory prediction, which integrates Dynamic Bayesian network and Sequence-to-Sequence model through an adaptive online weighting method. Dynamic Bayesian network utilizes environmental features and kinematic information to infer the pedestrian's motion intentions through probabilistic reasoning. Sequence-to-Sequence model views trajectory predictions as sequence generation tasks, in which the future trajectories are generated relying on the observed trajectories. A real-world pedestrian motion dataset is employed for model validations and it is also enlarged through data augmentation techniques to enable training of data-driven approaches. We compare our model with several typical baselines methods and results show that our model outperforms those baselines. The average error and the final destination error with one-second prediction are 0.04m and 0.10m in crossing scenarios, and 0.06m and 0.17m in stopping scenarios, respectively. The study expects to provide guidelines for the decision-making of intelligent vehicles in order to protect vulnerable road users.

Pedestrian trajectory prediction in urban environments has emerged as a critical research area with extensive applications across various domains. Accurate prediction of pedestrian trajectories is essential for the safe navigation of autonomous vehicles and robots in pedestrian-populated environments. Effective prediction models must capture both the spatial interactions among pedestrians and the temporal dependencies governing their movements. Existing research primarily focuses on forecasting a single trajectory per pedestrian, limiting its applicability in real-world scenarios characterised by diverse and unpredictable pedestrian behaviours. To address these challenges, this paper introduces the Graph Convolutional Network, Spatial–Temporal Attention, and Generative Model (GSTGM) for pedestrian trajectory prediction. GSTGM employs a spatiotemporal graph convolutional network to effectively capture complex interactions between pedestrians and their environment. Additionally, it integrates a spatial–temporal attention mechanism to prioritise relevant information during the prediction process. By incorporating a time-dependent prior within the latent space and utilising a computationally efficient generative model, GSTGM facilitates the generation of diverse and realistic future trajectories. The effectiveness of GSTGM is validated through experiments on real-world scenario datasets. Compared to the state-of-the-art models on benchmark datasets such as ETH/UCY, GSTGM demonstrates superior performance in accurately predicting multiple potential trajectories for individual pedestrians. This superiority is measured using metrics such as Final Displacement Error (FDE) and Average Displacement Error (ADE). Moreover, GSTGM achieves these results with significantly faster processing speeds.

Pedestrian trajectory prediction is important in various applications such as driverless vehicles, social robots, intelligent tracking systems and space planning. Existing methods focus on analysing the influence of neighbours but ignore the effect of the intended destinations of pedestrians which also plays a key role in route planning. In this paper, we propose a novel two- stage trajectory prediction method to yield multiple prediction trajectories with different probabilities towards different destination regions in the scene. Our method, which we refer to as Bi-Prediction, uses a bidirectional LSTM architecture to automatically classify trajectories into a small number of route classes before trajectory prediction. We have evaluated our method against two baseline methods and three state-of-art methods on two benchmark datasets. Our experimental results show that the extra classification stage improves the accuracy of the predicted trajectories.

Future pedestrian trajectory prediction offers great prospects for many practical applications such as unmanned vehicles, building evacuation design and robotic path planning. Most existing methods focus on social interaction among pedestrians but ignore the fact that heterogeneous traffic objects (cars, dogs, bicycles, motorcycles, etc.) have significant influence on the future trajectory of a subject pedestrian. Also, the walking direction intention of a pedestrian may be referred by his/her facial keypoints. Considering this, this work proposes to predict a pedestrian's future trajectory by jointly using neighboring heterogeneous traffic information and his/her facial keypoints. To fulfill this, an end-to-end facial keypoints-based convolutional encoder-decoder network (FK-CEN) is designed, in which the heterogeneous traffic and facial keypoints are input. After training, FK-CEN is evaluated on 5 crowded video sequences collected from the public datasets MOT-16 and MOT-17. Experimental results demonstrate that it outperforms state-of-the-art approaches, in terms of prediction errors.