SDAPNet: End-to-End Multi-task Simultaneous Detection and Prediction Network

Multi-modal data fusion plays a critical role in enhancing the accuracy and robustness of perception systems for autonomous driving, especially for the detection of small objects. However, small object detection remains particularly challenging due to sparse LiDAR points and low-resolution image features, which often lead to missed or imprecise detections. Currently, many methods process LiDAR point clouds and visible-light camera images separately, and then fuse them in the detection head. However, these approaches often fail to fully exploit the advantages of multi-modal sensors and overlook the potential for enhancing the correlation between modalities before feature fusion. To address this, we propose a novel LiDAR-guided multi-modal fusion framework for object detection, called LGMMfusion. This framework leverages the depth information from LiDAR to guide the generation of image Bird's Eye View (BEV) features. Specifically, LGMMfusion promotes spatial interaction between point clouds and pixels before the fusion of LiDAR BEV and image BEV features, enabling the generation of higher-quality image BEV features. To better align image and LiDAR features, we incorporate a multi-head multi-scale self-attention mechanism and a multi-head adaptive cross-attention mechanism, using the prior depth information from point clouds to generate image BEV features that better match the spatial positions of LiDAR BEV features. Finally, the LiDAR BEV features and image BEV features are fused to provide enhanced features for the detection head. Experimental results show that LGMMfusion achieves 71.1% NDS and 67.3% mAP on the nuScenes validation set, while also improving the detection of small objects and enhancing the detection accuracy of most objects.

Voxel is one of the common structural representation of 3D point cloud. Due to the sparsity of point cloud generated by light detection and ranging (LiDAR), there is the extreme imbalance in the foreground and background voxels. It decreases the accuracy of 3D object detection, has the negative effect on intelligent driving safety. To overcome this problem, we present a saliency prediction based 3D object detector SP-Det in this article. Although foreground voxels have the sufficient feature of object, it is difficult to localize the foreground region from voxel space with the larger background region. We design an auxiliary learning task, saliency prediction (SP). It benefits 3D detector in identifying the foreground region. SP task uses label diffusion to alleviate the label imbalance. It reduces the learning difficulty of saliency in voxel and bird's eye view (BEV) spaces. After that, to strengthen feature interaction from the sparse foreground region, we design saliency fusion (SF) module to fuse the learning result in SP task. It utilizes voxel and BEV saliency maps as progressive attention to resist the redundant feature from background region. To aggregate more foreground feature inside 3D and BEV region of interest (RoI), we design hybrid grid maps based RoI pooling (Hybrid-RoI pooling). Experiments are conducted in STF dataset. The adverse weather enlarges the sparsity of LiDAR point cloud, increasing the difficulty of object detection. SP-Det identifies and leverages foreground region, and achieves the performance better than the current methods. Hence, we believe that SP-Det benefits to LiDAR based 3D scene understanding in the adverse weather.

This paper describes new vision-based registration methods utilizing not only cameras on a user's head-mounted display but also a bird's-eye view camera that observes the user from an objective viewpoint. Two new methods, the line constraint method (LCM) and global error minimization method (GEM), are proposed. The former method reduces the number of unknown parameters concerning the user's viewpoint by restricting it to be on the line of sight from the bird's-eye view. The other method minimizes the sum of errors, which is the sum of the distance between the fiducials on the view and the calculated positions of them based on the current viewing parameters, for both the user's view and the bird's-eye view. The methods proposed here reduce the number of points that should be observed from the user's viewpoint for registration, thus improving the stability. In addition to theoretical discussions, this paper demonstrates the effectiveness of our methods by experiments in comparison with methods that use only a user's view camera or a bird's-eye view camera.

3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suitable for efficient parallelization. Secondly, they have a variable runtime due to level of sparsity of the scene which is against the determinism needed in a safety system. In this work, we aim to develop a very low latency algorithm with fixed runtime. We propose a novel semantic segmentation architecture as a single unified model for object center detection using key points, box predictions and orientation prediction using binned classification in a simpler Bird's Eye View (BEV) 2D representation. The proposed architecture can be trivially extended to include semantic segmentation classes like road without any additional computation. The proposed model has a latency of 4 ms on the embedded Nvidia Xavier platform. The model is 5X faster than other top accuracy models with a minimal accuracy degradation of 2% in Average Precision at IoU=0.5 on KITTI dataset.

The Medieval Church of St. Andrews. Edited by David McRoberts. Pp. xii + 178 + 24 illustrations + plan and a facsimile of the 16th-century ‘bird's eye view’ of St. Andrews. Glasgow: Burns, 1976. £7. - Volume 29 Issue 3

Our research objective is construction of 3D map and development of its practical applications. As an application, the authors had proposed 3D control interface for an operator who controls remote control robot. It is well know that an operator can control a robot easily using bird's-eye view. Using 3D control interface, an operator can control the robot at bird's-eye view. The bird's-eye view is constructed from 3D map and robot model. The operator can recognize relationship between the robot and the environment from 3D map and robot model following real robot motion. In this paper, we will describe 3D control interface and its implementation for HELIOS carrier. We will show an experimental result of 3D control interface. From result, we confirmed the effectiveness of the proposed method.

Robotics combines a lot of different domains with sophisticated challenges such as computer vision, motion control and search algorithms. Search algorithms can be applied to calculate movements. The A∗ algorithm is a well-known and proved search algorithm to find a path within a graph. This paper presents an extended A∗ algorithm that is optimized for robot navigation using a bird's eye view as a map that is dynamically generated by image stitching. The scenario is a robot that moves to a target in an environment containing obstacles. The robot is controlled by a Xilinx Zynq platform that contains an ARM processor and an FPGA. In order to exploit the flexibility of such an architecture, the FPGA is used to execute the most compute-intensive task of the extended A∗ algorithm. This task is responsible for sorting the accessible nodes in the graph. Several environments with different complexity levels are used to evaluate the extended A∗ algorithm. The environment is captured by a Kinect sensor located directly on the robot. In order to dewarp the robot's view, the frames are transformed to a bird's eye view. In addition, a wider viewing range is achieved by image stitching. The evaluation of the extended A∗ algorithm shows a significant improvement in terms of memory utilization. Accordingly, this algorithm is especially practicable for embedded systems since they have often only limited memory resources. Moreover, the overall execution time for several use cases is reduced up to a speed-up of 2.88x.

Bird's Eye View (BEV) representation learning has greatly improved target detection and tracking in complex environments and autonomous systems, thanks to a multi-view aggregation strategy that enhances spatial awareness, reduces occlusions, and ensures continuous surveillance of targets. This paper presents an Enhanced Global Context Fusion Network for multi-target multi-camera tracking (MTMC), addressing the critical challenge of spatial association across views in large-scale surveillance systems, unlike conventional approaches that directly aggregate Bird's Eye View (BEV) features, our method hierarchically refines individual BEV embedding by integrating global context, generating bootstrap features that better preserve spatial relationships and reduce information loss. In order to improve the problem of poor robustness to real-world issues that may be caused by camera malfunctions, we introduce a data augmentation strategy that replaces the input of lost camera signals with black images during training. The results of extensive experiments on the Wildtrack dataset demonstrate the state-of-the-art performance of our approach, reaching 92.0% for MODA and 94.1% for IDF1, outperforming existing techniques with only a negligible increase in computational cost. These results validate the effectiveness of our approach in terms of improved BEV feature representation and system stability, providing a reliable solution for reliable MTMC detection and tracking in complex environments.

By matching roadway images in encountered scene with bird's eye view, the scope of humans' perception is expanded to satellite-roadway-vehicle network. Based on the geometric consistency of the satellite images with as-is local terrain, in this paper, a computa- tional mechanism is introduced for generation and adaptation of a roadway model transferable through the network. First, the chromatic complexity of the roadway area is represented as a palette of saliency colors via fractal sampling of the scene image. Next, the palette is adapted to the associated area in the bird's eye view. Finally, the palette is transferred to the bird's eye view for anticipatively following the roadway pattern. Experimental results demonstrate that the transferable model can be applied to the extension of the roadway pattern prior to physical access.

[This corrects the article DOI: 10.1098/rsos.160426.].

This work offers a method for the visual-based navigation of a mobile robot in indoor environments, using a single omni-directional camera. The geometry of the catadioptric sensor and the method used to obtain a bird's eye (orthographic) view of the ground plane are presented. This picture evocatively abridges the solution to steering problems, by eliminating any perspective effects. The nature of each navigation task is taken hooked on account when conniving the compulsory navigation skills and environmental representations. Topological Navigation is cast-off for itinerant long distances plus does not require knowledge of the exact location of the robot but rather, a qualitative position happening the topological map. For indoor robots in structured environments, we consume owed separately with the cases of geometrical and topological models of interplanetary. For shapeless environments, we consume discussed the cases of navigation through optical activities, using methods from the appearance-based paradigm, and by recognition of specific substances in the environment. The navigation process cartels appearance based approaches and visual serving upon some environmental features. Graphic Path Succeeding is obligatory for native, actual precise navigation, for e.g. door traversal, docking. The robot is controlled to follow a pre-specified trail accurately, by tracking visual milestones in bird's eye views of the ground plane. By clearly separating the nature of these triangulation tasks, a simple besides yet powerful navigation scheme is obtained.

Proposes a method for the visual-based navigation of a mobile robot in indoor environments, using a single omnidirectional (catadioptric) camera. The geometry of the catadioptric sensor and the method used to obtain a bird's eye (orthographic) view of the ground plane are presented. This representation significantly simplifies the solution to navigation problems, by eliminating any perspective effects. The nature of each navigation task is taken into account when designing the required navigation skills and environmental representations. We propose two main navigation modalities: topological navigation and visual path following. Topological navigation is used for traveling long distances and does not require knowledge of the exact position of the robot but rather, a qualitative position on the topological map. The navigation process combines appearance based methods and visual servoing upon some environmental features. Visual path following is required for local, very precise navigation, e.g., door traversal, docking. The robot is controlled to follow a prespecified path accurately, by tracking visual landmarks in bird's eye views of the ground plane. By clearly separating the nature of these navigation tasks, a simple and yet powerful navigation system is obtained.

This paper reports an experiment, conducted with 116 undergraduates at an institute of Higher Education in the United Kingdom, to determine if preferences exist in the way selected landscape features and the water (hydrological) cycle are portrayed. The results show that most students can draw simple diagrams of physical landforms from descriptions, but there is some confusion between bird's eye and side views. A significant number of students draw bird's eye views of drainage basins which flow downwards and coastlines which trend from east to west. In sketches of the side views of cliffs, waterfalls, river profiles and landslips, a significant number of students place the feature on the left, facing towards the right. No significant differences exist in the orientation of side views of the water cycle. It is proposed that the preferred orientations are explained by research findings into stroke patterns, although other factors are also identified to explain the results.

Driving requires interacting with road agents and predicting their future behaviour in order to navigate safely. We present FIERY: a probabilistic future prediction model in bird's-eye view from monocular cameras. Our model predicts future instance segmentation and motion of dynamic agents that can be transformed into non-parametric future trajectories. Our approach combines the perception, sensor fusion and prediction components of a traditional autonomous driving stack by estimating bird's-eye-view prediction directly from surround RGB monocular camera inputs. FIERY learns to model the inherent stochastic nature of the future solely from camera driving data in an end-to-end manner, without relying on HD maps, and predicts multimodal future trajectories. We show that our model outperforms previous prediction baselines on the NuScenes and Lyft datasets. The code and trained models are available at https://github.com/wayveai/fiery.

Accurate object detection is a fundamental requirement for autonomous systems to operate in dynamic urban environments. Considering the intricacy of the environment and the occlusion of objects, various point cloud based three-dimensional (3D) object detection methods have been proposed, such as point-based or voxel-based methods. In this paper, a feature selection mechanism is proposed in a voxel-based method to generate bird’s eye view (BEV) feature maps from the original point cloud. For the 3D Backbone, the U-shaped structure and single scale feature selection module are combined. After combining high-level semantics and low-level fine-grained features, the optimized features after the BEV feature maps are applied to region of interest (RoI) refinement, so that the voxel features can better serve the subsequent 3D object detection. The experimental results on KITTI dataset show higher 3D object detection accuracy compared to the state-of-the-art 3D detection methods, which reflects the effectiveness of the proposed architecture.