Intention-Interaction Graph Based Hierarchical Reasoning Networks for Human Trajectory Prediction

Pedestrian trajectory prediction is a critical research area with numerous domains, e.g., blind navigation, autonomous driving systems, and service robots. There exist two challenges in this research field: spatio-temporal interaction modeling among pedestrians and the uncertainty of pedestrian trajectories. To tackle these challenges, we propose a spatio-temporal interaction aware and trajectory distribution aware graph convolution network. First, we propose a spatio-temporal interaction aware module that integrates a graph convolutional network and self-attention mechanism to model spatio-temporal interactions among pedestrians. Second, we design a trajectory distribution aware module to learn latent trajectory distribution information from the measured trajectories at observed and future times. This can provide knowledge-rich trajectory distribution information for the multimodality of the predicted trajectories. Finally, to address the problem of the propagation and accumulation of prediction errors, we design a trajectory decoder to generate the multimodal future trajectories. The proposed model is evaluated utilizing videos recorded by a camera sensor in crowded areas and can be applied to predict multiple pedestrians’ future trajectories from in-vehicle cameras. Experimental results demonstrate that the proposed approach can achieve superior results on the average displacement error (ADE) and final displacement error (FDE) metrics to state-of-the-art approaches and can predict socially acceptable future trajectories.

Pedestrian trajectory prediction is fundamental to a wide range of scientific research work and industrial applications. Most of the current advanced trajectory prediction methods incorporate context information such as pedestrian neighbourhood, labelled static obstacles, and the background scene into the trajectory prediction process. In contrast to these methods which require rich contexts, the method in our paper focuses on predicting a pedestrian's future trajectory using his/her observed part of the trajectory only. Our method, which we refer to as LVTA, is a Location-Velocity-Temporal Attention LSTM model where two temporal attention mechanisms are applied to the hidden state vectors from the location and velocity LSTM layers. In addition, a location-velocity attention layer embedded inside a tweak module is used to improve the predicted location and velocity coordinates before they are passed to the next time step. Extensive experiments conducted on three large benchmark datasets and comparison with eleven existing trajectory prediction methods demonstrate that LVTA achieves competitive prediction performance. Specifically, LVTA attains 9.19 pixels Average Displacement Error (ADE) and 17.28 pixels Final Displacement Error (FDE) for the Central Station dataset, and 0.46 metres ADE and 0.92 metres FDE for the ETH&UCY datasets. Furthermore, evaluation on using LVTA to generate trajectories of different prediction lengths and on new scenes without the need of retraining confirms that it has good generalizability.

Non-probability sampling network based on anomaly pedestrian trajectory discrimination for pedestrian trajectory prediction

Future pedestrian trajectory prediction offers great prospects for many practical applications such as unmanned vehicles, building evacuation design and robotic path planning. Most existing methods focus on social interaction among pedestrians but ignore the fact that heterogeneous traffic objects (cars, dogs, bicycles, motorcycles, etc.) have significant influence on the future trajectory of a subject pedestrian. Also, the walking direction intention of a pedestrian may be referred by his/her facial keypoints. Considering this, this work proposes to predict a pedestrian's future trajectory by jointly using neighboring heterogeneous traffic information and his/her facial keypoints. To fulfill this, an end-to-end facial keypoints-based convolutional encoder-decoder network (FK-CEN) is designed, in which the heterogeneous traffic and facial keypoints are input. After training, FK-CEN is evaluated on 5 crowded video sequences collected from the public datasets MOT-16 and MOT-17. Experimental results demonstrate that it outperforms state-of-the-art approaches, in terms of prediction errors.

Pedestrian behavior anticipation is a key challenge in the design of assistive and autonomous driving systems suitable for urban environments. An intelligent system should be able to understand the intentions or underlying motives of pedestrians and to predict their forthcoming actions. To date, only a few public datasets were proposed for the purpose of studying pedestrian behavior prediction in the context of intelligent driving. To this end, we propose a novel large-scale dataset designed for pedestrian intention estimation (PIE). We conducted a large-scale human experiment to establish human reference data for pedestrian intention in traffic scenes. We propose models for estimating pedestrian crossing intention and predicting their future trajectory. Our intention estimation model achieves 79% accuracy and our trajectory prediction algorithm outperforms state-of-the-art by 26% on the proposed dataset. We further show that combining pedestrian intention with observed motion improves trajectory prediction. The dataset and models are available at http://data.nvision2.eecs.yorku.ca/PIE_dataset/.

Pedestrian trajectory prediction via physical-guided position association learning

Pedestrian trajectory prediction can provide decision support for self-driving cars and robots to avoid potential collision risks. Modelling of social relationships and multimodal prediction are core challenges in pedestrian trajectory prediction. However, there are two problems with existing methods: (1) these methods usually focus on modelling pairwise or group interactions, ignoring the importance of high-order interactions; and (2) existing methods struggle to balance the performance between deterministic and multimodal predictions. To address these issues, we propose a higher-order graph attention network for pedestrian trajectory prediction. Specifically, we first construct spatial and temporal graphs to model social interactions and temporal dependencies. To achieve high-order interaction modelling, we design a higher-order graph attention module to capture pedestrian social and temporal interactions of different orders by stacking multiple layers of graph attention. Subsequently, we design a deformable spatial-temporal sparse module to adaptively eliminate social and temporal redundant interactions. To balance deterministic and multimodal prediction performance, we design a two-stage prediction strategy. The first stage predicts pedestrian deterministic trajectories, and the second stage refines pedestrian deterministic trajectories to complete both deterministic and multimodal predictions. Quantitative analysis on the ETH, UCY, and SDD datasets demonstrates that our method outperforms existing methods in both deterministic and multimodal predictions. Qualitative analysis shows that our method can effectively describe complex social relationships and accomplish reliable trajectory prediction.

Forecasting pedestrian trajectories in dynamic scenes remains a critical problem in various applications, such as autonomous driving and socially aware robots. Such forecasting is challenging due to human-human and human-object interactions and future uncertainties caused by human randomness. Generative model-based methods handle future uncertainties by sampling a latent variable. However, few studies explored the generation of the latent variable. In this work, we propose the trajectory predictor with pseudo Oracle (TPPO), which is a generative model-based trajectory predictor. The first pseudo oracle is pedestrians’ moving directions, and the second one is the latent variable estimated from ground truth trajectories. A social attention module is used to aggregate neighbors’ interactions based on the correlation between pedestrians’ moving directions and future trajectories. This correlation is inspired by the fact that pedestrians’ future trajectories are often influenced by pedestrians in front. A latent variable predictor is proposed to estimate latent variable distributions from observed and ground-truth trajectories. Moreover, the gap between these two distributions is minimized during training. Therefore, the latent variable predictor can estimate the latent variable from observed trajectories to approximate that estimated from ground-truth trajectories. We compare the performance of TPPO with related methods on several public datasets. Results demonstrate that TPPO outperforms state-of-the-art methods with low average and final displacement errors. The ablation study shows that the prediction performance will not dramatically decrease as sampling times decline during tests.

This paper develops a pedestrian potentially dangerous behaviour prediction method based on attention‐long‐short‐term memory (Attention‐LSTM) architecture to predict pedestrian trajectory and intention for the unexpected pedestrian crossing accident avoidance. To extract the road scene information for short periods of time, and improve the accuracy of subsequent intention inference and trajectory prediction, the panoramic segmentation is used to extrapolate pedestrian instances and segment areas of the environment. Next, an encoder–decoder framework based on Attention‐LSTM model is proposed to infer a pedestrian's intention to run or walk out into oncoming traffic straight and to predict the future trajectory. The proposed network involves two parts: temporal feature encoder and multi‐task decoder. The temporal feature encoder is mainly used to selectively emphasize the temporal features using attention mechanism, and then LSTM is employed for its encoding. In the multi‐task decoder, a multi‐head self‐attention mechanism and LSTM are used to forecast the pedestrians’ intention and future trajectory, respectively. Extensive experiments on pedestrian intention estimation (PIE) datasets demonstrate that the authors’ proposed approach surpasses prior studies in terms of prediction accuracy in trajectory and intention. This study can not only effectively avoid serious accidents caused by illegal road crossing but also achieve early warning for collision avoidance.

Trajectory prediction is a challenging problem in autonomous driving, due to changes in scene structure and multimodal distributions of future trajectories. This study proposes a novel trajectory prediction model (LGINet). The central innovation of LGINet is the Context-Aware Graph Attention (CAGA) layer, which achieves a balance between handling scene complexity and the ability to capture information, improving prediction performance. Unlike conventional attention mechanisms, CAGA uses Sparsemax to focus on key neighbors, reducing noise and enhancing computational efficiency. The multi-head attention mechanism enhances the ability of the model to capture interactions across both local and long-distance spatial scales. Experiments with the nuScenes dataset show that the proposed LGINet model outperforms existing prediction methods. Specifically, the MinADE 5 value of LGINet is 1.23, the MR 5,2 value is 0.49, and the MinFDE 5 value is 2.39, demonstrating its effectiveness in generating diverse and accurate future trajectories for autonomous driving applications. Qualitative comparisons confirm that LGINet generates diverse trajectories aligned with scene structures and agent motion, highlighting its potential for autonomous driving.

Accurate trajectory prediction of surrounding road users is critical to autonomous driving systems. In mixed traffic flows, road users with different kinds of behaviors and styles bring complexity to the environment, which requires considering interactions among road users when anticipating their future trajectories. This paper presents a long-term interactive trajectory prediction method for surrounding vehicles using a hierarchical multi-sequence learning network. In contrast to non-interactive method which assumes that road users are independent of each other, this method can automatically learn high-level dependencies among multiple interacting vehicles through the proposed structural-LSTM (long short-term memory) network. Specifically, structural-LSTM first assigns one LSTM for each interacting vehicle. Then these LSTMs share their cell states and hidden states with their spatial-neighboring LSTMs by a radial connection, and recurrently analyze the output state of itself as well as the other LSTMs in a deeper layer. Finally based on all output states, the network predicts trajectories for surrounding vehicles. The proposed method is evaluated on the NGSIM dataset, and its results show that satisfyingly accurate prediction performance of long-term trajectories of surrounding vehicles is accessible, e.g., longitudinal and lateral RMS error can be reduced to less than 1.93m and 0.31m over 5s time horizon, respectively.

Physics constrained pedestrian trajectory prediction with probability quantification

Pedestrian behavior prediction is essential to enable safe and efficient driving of intelligent vehicles on urban traffic environment. This article presents a novel framework for pedestrian trajectory prediction, which integrates Dynamic Bayesian network and Sequence-to-Sequence model through an adaptive online weighting method. Dynamic Bayesian network utilizes environmental features and kinematic information to infer the pedestrian's motion intentions through probabilistic reasoning. Sequence-to-Sequence model views trajectory predictions as sequence generation tasks, in which the future trajectories are generated relying on the observed trajectories. A real-world pedestrian motion dataset is employed for model validations and it is also enlarged through data augmentation techniques to enable training of data-driven approaches. We compare our model with several typical baselines methods and results show that our model outperforms those baselines. The average error and the final destination error with one-second prediction are 0.04m and 0.10m in crossing scenarios, and 0.06m and 0.17m in stopping scenarios, respectively. The study expects to provide guidelines for the decision-making of intelligent vehicles in order to protect vulnerable road users.

Multi-agent trajectory prediction plays a pivotal role for intelligent transportation and autonomous driving. Modeling the social interaction among agents and revealing the inherent relationship between interaction and future trajectory are crucial for accurate trajectory prediction. To address these challenges, this letter proposes a goal-guided and interaction-aware state refinement graph attention network (SRGAT) for multi-agent trajectory prediction. Specifically, the trajectory sequence of individual agent is firstly converted into a compact representation by exploiting Transformer encoder and gated recurrent unit (GRU). Then, a social state refinement (SSR) module for modeling social influence between agents is proposed to include more interaction-related features. Subsequently, multiple goals are predicted for each agent, followed by another goal-guided SSR module to incorporate goal information into social interaction. Finally, the multimodal trajectory is forecasted by fusing the features from forward and backward GRU. Experiments on public benchmark datasets are carried out to evaluate the effectiveness of our model. The results demonstrate the superior performance of our model compared with the state-of-the-art methods.

This work remedies the drawbacks of existing methods in pedestrian trajectory prediction for video surveillance. A traditional two-stage approach, which first detects and tracks pedestrians before forecasting their subsequent trajectories, relies solely on historical trajectories to predict future movements, overlooking the rich contextual information within videos. Although some one-stage methods have been introduced to streamline this process, they still struggle with complex model architectures due to their reliance on pixel-wise flow estimation. To resolve these issues, we propose a simple yet effective Transformer-based one-stage model, called PredDETR, that directly anticipates the future trajectories of multiple pedestrians from videos. Following the end-to-end paradigm of DETR, our PredDETR model not only associates the bounding boxes of each pedestrian across subsequent frames but also predicts their future trajectories using an additional predictive decoder. Taking past raw video frames as input, the proposed PredDETR directly forecasts the future locations of each pedestrian in a non-autoregressive manner. The empirical results validate that, despite its simpler design, PredDETR is a compelling method compared to a previous one-stage approach.