Multimodal multi-pedestrian path prediction for autonomous cars

Accurate prediction of the future position of pedestrians in traffic scenarios is required for safe navigation of an autonomous vehicle but remains a challenge. This concerns, in particular, the effective and efficient multimodal prediction of most likely trajectories of tracked pedestrians from egocentric view of self-driving car. In this paper, we present a novel solution, named M2P3, which combines a conditional variational autoencoder with recurrent neural network encoder-decoder architecture in order to predict a set of possible future locations of each pedestrian in a traffic scene. The M2P3 system uses a sequence of RGB images delivered through an internal vehicle-mounted camera for egocentric vision. It takes as an input only two modes, that are past trajectories and scales of pedestrians, and delivers as an output the three most likely paths for each tracked pedestrian. Experimental evaluation of the proposed architecture on the JAAD and ETH/UCY datasets reveal that the M2P3 system is significantly superior to selected state-of-the-art solutions.

Memory degradation induced by attention in recurrent neural architectures

The human brain processes enormous volumes of high-dimensional data for everyday perception. To humans, a picture is worth a thousand words, but to a machine, it is just a seemingly random array of numbers. Although machines are very fast and efficient, they are vastly inferior to humans for everyday information processing. Algorithms that mimic the way the human brain computes and learns may be the solution. We present a theoretical model based on the observation that images of similar visual perceptions reside in a complex manifold in a low-dimensional image space. The perceived features are often highly structured and hidden in a complex set of relationships or high-dimensional abstractions. To model the pattern manifold, we present a novel learning algorithm using a recurrent neural architecture. The brain memorizes information using a dynamical system made of interconnected neurons. Retrieval of information is accomplished in an associative sense. It starts from an arbitrary state that might be an encoded representation of a visual image and converges to another state that is stable. The stable state is what the brain remembers. In designing a recurrent neural architecture, it is usually of prime importance to guarantee the convergence in the dynamics of the network. We propose to modify this picture: if the brain remembers by converging to the state representing familiar patterns, it should also diverge from such states when presented with an unknown encoded representation of a visual image belonging to a different category. That is, the identification of an instability mode is an indication that a presented pattern is far away from any stored pattern and therefore cannot be associated with current memories. These properties can be used to circumvent the plasticity-stability dilemma by using the fluctuating mode as an indicator to create new states. We capture this behavior using a novel neural architecture and learning algorithm, in which the system performs self-organization utilizing a stability mode and an instability mode for the dynamical system. Based on this observation we developed a self-organizing line attractor, which is capable of generating new lines in the feature space to learn unrecognized patterns. Experiments performed on various databases show that the proposed nonlinear line attractor is able to successfully associate patterns and it provides better association when compared to other state of the art techniques. It shows that the proposed model is able to create nonlinear manifolds in a multidimensional feature space to distinguish complex patterns.

Semantic segmentation technology in traffic scenes can help vehicles make accurate analyses and positioning of the road ahead. In traffic scenarios, the trade-off between real-time performance and the accuracy of semantic segmentation is particularly important. This paper proposes a lightweight deep convolutional network, which can be applied to traffic scenes to complete accurate semantic segmentation tasks, considering both real-time performance and accuracy. The Cross Channel Attention Fusion Mechanism proposed in this paper can better integrate the context information and improve accuracy. The Depth-wise Separable Pyramid Module proposed based on the feature pyramid idea can improve the segmentation accuracy and effectively trade off the real-time performance.

At the current stage, the rapid Development of autonomous driving has made object detection in traffic scenarios a vital research task. Object detection is the most critical and challenging task in computer vision. Deep learning, with its powerful feature extraction capabilities, has found widespread applications in safety, military, and medical fields, and in recent years has expanded into the field of transportation, achieving significant breakthroughs. This survey is based on the theory of deep learning. It systematically summarizes the Development and current research status of object detection algorithms, and compare the characteristics, advantages and disadvantages of the two types of algorithms. With a focus on traffic signs, vehicle detection, and pedestrian detection, it summarizes the applications and research status of object detection in traffic scenarios, highlighting the strengths, limitations, and applicable scenarios of various methods. It introduces techniques for optimizing object detection algorithms, summarizes commonly used object detection datasets and traffic scene datasets, along with evaluation criteria, and performs comparative analysis of the performance of deep learning algorithms. Finally, it concludes the development trends of object detection algorithms in traffic scenarios, providing research directions for intelligent transportation and autonomous driving.

Accurate identification of road object is crucial for intelligent traffic systems. However, due to the complexity of road traffic scenarios, developing efficient and accurate road object detection methods has been a challenging task. In this study, a new improved method for road object detection is proposed, named Enhanced YOLOv5 algorithm and Deep Schedule Object Detection (DSOD) algorithm. Real traffic scenes from the BDD100k dataset are used for training and testing the object detection model. The dataset consists of 9 different types of road objects in various traffic scenarios. The Mosaic data augmentation algorithm is applied to merge images in the dataset. Mean Average Precision (mAP), Precision (P), and Recall (R) are used as evaluation metrics to compare the enhanced YOLOv5 model with the most common models. Experimental results demonstrate that the DSOD algorithm achieves success in intelligent traffic object detection, significantly improving the accuracy and robustness of road object recognition. Additionally, the developed model shows significant performance improvement in accurately identifying objects in complex traffic scenes. These results suggest that the DSOD algorithm is a promising choice for intelligent road recognition and can easily adapt to different traffic scenarios. Furthermore, employing cloud computing for real-time detection meets the requirements of intelligent cooperative vehicles and enhances their visual perception capabilities.

Accurately predicting the trajectory of road agents in complex traffic scenarios is challenging because the movement patterns of agents are complex and stochastic, not only depending on their own past trajectories but also being closely related to the social interaction with other surrounding agents. Besides accuracy, efficient prediction with low inference latency is also a highly desirable feature for the practical application of trajectory prediction. To address these issues, we propose the VNAGT model, a variational non-autoregressive graph transformer that adopts the framework of conditional variational autoencoder and incorporates the non-autoregressive approach such that diverse trajectories and low prediction latency can be achieved simultaneously. In order to capture the social and temporal interaction, we put forward a unified graph attention-based module that is applicable for homogeneous and heterogeneous multi-agents such that the class information can be seamlessly integrated when it is available. Non-autoregressive decoding is combined with variational learning to produce multiple plausible predictions with low latency. We train and validate the model on two real-world homogeneous and heterogeneous trajectory datasets. The experimental results demonstrate the superior performance of our model in comparison with the state-of-the-art methods.

When predicting future motions of surrounding vehicles for autonomous vehicles, the inter-vehicular interaction must be considered in order to predict future risks and to make safe and intelligent decisions. This becomes critical when it comes to conflicting driving situation such as lane merge, tollgate area, and unsignalized intersections. Previously developed future prediction algorithms show limited performance when handling interactions and conflicts between vehicles because they focused on predicting individual vehicle motion and/or interaction between a single pair of vehicles rather than the entire traffic scene. In this paper, a scene representation method, namely multi-channel Occupancy Grid Map (OGM), is proposed to describe the entire traffic scene, which is then utilized for the deep learning architecture that predicts the future traffic scene or OGM. Multi-channel OGM represents entire traffic scene as a manner of image-like structure from bird’s eye view composed with dynamic layer and static layer depicting the occupancy of the dynamic and static objects. By using this 2D traffic scene representation, future prediction can be modeled as a video processing problem, where future time-serial image sequence need to be predicted. In order to predict future traffic scenes based on past traffic scenes, a deep learning architecture is proposed using Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) Networks. With the proposed deep learning architecture, future prediction accuracy in highly conflicting traffic situation is guaranteed up to 90 percent with 3 seconds of prediction horizon. A video ofthe traffic scene prediction results is available online [1].

Previous camera self-calibration methods have exhibited certain notable shortcomings. On the one hand, they either exclusively emphasized scene cues or solely focused on vehicle-related cues, resulting in a lack of adaptability to diverse scenarios and a limited number of effective features. Furthermore, these methods either solely utilized geometric features within traffic scenes or exclusively extracted semantic information, failing to comprehensively consider both aspects. This limited the comprehensive feature extraction from scenes, ultimately leading to a decrease in calibration accuracy. Additionally, conventional vanishing point-based self-calibration methods often required the design of additional edge-background models and manual parameter tuning, thereby increasing operational complexity and the potential for errors. Given these observed limitations, and in order to address these challenges, we propose an innovative roadside camera self-calibration model based on the Transformer architecture. This model possesses a unique capability to simultaneously learn scene features and vehicle features within traffic scenarios while considering both geometric and semantic information. Through this approach, our model can overcome the constraints of prior methods, enhancing calibration accuracy and robustness while reducing operational complexity and the potential for errors. Our method outperforms existing approaches on both real-world dataset scenarios and publicly available datasets, demonstrating the effectiveness of our approach.

Target detection in traffic scenes has been a focal point concerning computer vision, especially in the context of autonomous driving, including the recognition of vehicles, pedestrians, and traffic signs. However, the computational power of devices operating in traffic scenarios is limited, placing stringent demands on computational resources and latency. To address these challenges, this study proposed a lightweight detection algorithm, GV2-YOLO, which prioritizes both speed and accuracy. First, we apply the GhostNetv2 architecture and incorporate the Ghost module to decrease the parameters of the backbone feature extraction network. Second, the algorithm also integrates SPPF and Slim-neck by GSConv techniques, which effectively reduces the computational complexity while ensuring the resulting significant reduction in detection accuracy. Our proposed algorithm achieves a mAP of 84.7% on the CCTSDB dataset. The algorithm has a total parameter count of 9.1 M, making it ideal for deployment in embedded autonomous driving platforms.

This work proposes a bio-inspired neural architecture called L-PREEN (Learning and Perceptual boundaRy rEcurrent dEtection neural architecture). L-PREEN has three different feedback interactions that fuse the bottom-up and top-down contour information of visual areas V1-V2-V4-Infero Temporal. This recurrent model uses colour, texture, and diffusive features to generate surface perception and contour learning and recognition processes. We compare the L-PREEN model against other boundary detection methods using the Berkeley Segmentation Dataset and Benchmark (Martin et al., 2001). The results obtained show better performance of L-PREEN using quantitative measures.

Localization of a vehicle is a key component for driving assistance or autonomous navigation. In this work, we propose a visual positioning system (VPS) for vehicle or mobile robot navigation. Different from general landmark-based or model-based approaches, which rely on some predefined known landmarks or a priori information about the environment, no assumptions on the prior knowledge of the scene are made. A stereo-based vision system is built for both extracting feature correspondences and recovering 3-D information of the scene from image sequences. Relative positions of the camera motion are then estimated by registering the 3-D feature points from two consecutive image frames. Localization of the vehicle is finally given by the reference to its initial position. I. INTRODUCTION Localization and tracking of a vehicle are major compo- nents for providing positions, directions and travel informa- tion to the driver. They also serve as key technologies for building autonomous navigation systems. Many approaches have been proposed for locating a mobile robot or a ve- hicle based on various techniques. The most commonly used methods include dead-reckoning techniques, navigation using active beacons, landmark-based navigation, map-based navigation, global positioning system (GPS), and vision- based positioning. Dead-reckoning is a procedure for determining the present location of a vehicle by advancing some previous position through known path and velocity information over a given period of time. Since an odometer and optical sensors for wheel direction detection are easily installed on a vehicle, dead-reckoning is usually a less expensive method for vehicle localization. However, the integration of incremental motion information over time will lead to accumulation of errors. In the case of long distance travel, accumulation of orientation error will make position errors diverge as the increase of driving time. Active beacons such as laser, sonar or radio can be used as media for vehicle or mobile robot navigation. These approaches use triangulation to measure the distance between a number of beacons and the mobile platform and then deter- mine the current location. The problems associated with this technique are the inaccuracy of the distance measurement caused by time-delay of the signals, and the installation and maintenance cost of a large number of beacons required for an area. Satellite based differential global positioning system (DGPS) is probably the most advanced and accurate method for identifying the position and orientation of an object. However, DGPS does not work well if the satellite signals are blocked. This situation commonly happens in the indoor environments, or urban areas with tall buildings, etc. In this work, we propose a visual positioning system (VPS) for vehicle or mobile robot navigation. Different from general landmark-based or model-based approaches which rely on some predefined known landmarks or ap rioriinfor- mation about the environment (e.g., 3-D models of buildings, objects, etc.), no assumptions on the prior knowledge of the scene are made. A stereo-based vision system is built for both extracting feature correspondences and recovering 3- D information of the scene from image sequences. A robust feature tracking method based on simultaneously considering the inter-frame images from the same camera and the stereo image pair from different cameras at a fixed time instant is developed. The relative position of the camera motion is then estimated by registering the 3-D feature points from two consecutive image frames. Our system includes the following basic modules:

A Method of Vehicle Interactive Information Drive Speed Prediction Based on Temporal Dynamic Graph Convolutional Neural Network

Obstacle detection by evaluation of optical flow fields from image sequences

This Reprint presents a collection of cutting-edge research and review articles focusing on the integration of machine learning and deep learning theories for intelligent fault diagnosis in industrial and engineering systems. With the rapid advancement of computational intelligence, data-driven fault diagnosis has become a cornerstone of modern Prognostics and Health Management (PHM), enabling early detection, prediction, and mitigation of system failures.The contributions in this Reprint highlight innovative applications of artificial neural networks, convolutional and recurrent neural architectures, transfer learning, and hybrid intelligent systems for diagnosing faults in rotating machinery, robotic systems, power plants, and manufacturing processes. In addition, the included studies explore explainable AI models, data augmentation, and sensor fusion methods that enhance model interpretability and robustness under real-world operating conditions.By bringing together theoretical insights and practical implementations, this Reprint aims to serve as a valuable reference for researchers, engineers, and practitioners engaged in machine learningbased condition monitoring and intelligent fault diagnosis. It reflects recent trends shaping the future of autonomous and resilient industrial systems within the framework of Industry 4.0.