Toward Non-Linear, Fast, and Multi-Frame 3D Point Cloud Interpolation in Autonomous Driving

Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pretraining method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups for distilling self-supervised pre-trained image representations into 3D models. Hence, our method does not require any point cloud nor image annotations. The keyingredient of our method is the use of superpixels which are used to pool 3D point features and 2D pixel features in visually similar regions. We then train a 3D network on the self-supervised task of matching these pooled point features with the corresponding pooled image pixel features. The advantages of contrasting regions obtained by superpixels are that: (1) grouping together pixels and points of visually coherent regions leads to a more meaningful contrastive task that produces features well adapted to 3D semantic segmentation and 3D object detection; (2) all the different regions have the same weight in the contrastive loss regardless of the number of 3D points sampled in these regions; (3) it mitigates the noise produced by incorrect matching of points and pixels due to occlusions between the different sensors. Extensive experiments on autonomous driving datasets demonstrate the ability of our image-to-Lidar distillation strategy to produce 3D representations that transfer well on semantic segmentation and object detection tasks.

The article discusses the importance of selfdriving cars to improve road safety and reduce the number of accidents caused by human error. Self-driving cars not only reduce human error but also help reduce driver fatigue. We further explore the use of computer vision in autonomous cars, with previous research relying on deep learning algorithms with LiDAR sensors which can be expensive. The authors propose a more cost-effective approach using simple computer vision algorithms such as color space transformation, Canny edge detection, and Hough line transformation to detect lane lines and steer the car accordingly. This approach requires less operational hardware and can be implemented using affordable boards like Raspberry Pi and Nvidia Jetson Nano. The article also highlights the reconstruction of a remote-controlled car that had a 95% accuracy using a certain set of parameters was a tool for understanding autonomous cars better.

Nowadays, closed-set perception methods for autonomous driving perform well on datasets containing normal scenes. However, they still struggle to handle anomalies in the real world, such as unknown objects that have never been seen while training. The lack of public datasets to evaluate the model performance on anomaly and corner cases has hindered the development of reliable autonomous driving systems. Therefore, we propose a multimodal Synthetic Dataset for Anomaly and Corner case detection, called SDAC, which encompasses anomalies captured from multi-view cameras and the LiDAR sensor, providing a rich set of annotations for multiple mainstream perception tasks. SDAC is the first public dataset for autonomous driving that categorizes anomalies into object, scene, and scenario levels, allowing the evaluation under different anomalous conditions. Experiments show that closed-set models suffer significant performance drops on anomaly subsets in SDAC. Existing anomaly detection methods fail to achieve satisfactory performance, suggesting that anomaly detection remains a challenging problem. We anticipate that our SDAC dataset could foster the development of safe and reliable systems for autonomous driving.

In the recent era, a lot of interest is attracted by the autonomous vehicles which can sense surroundings and navigate without human intervention. Object detection and recognition form a major part of autonomous driving systems. Lidar sensors can be used to capture point clouds of driving environment. Detecting multiple 3D objects in point clouds in real time and defining their boundaries with the help of 3D bounding boxes are critical in motion planning by self driving systems. This paper proposes a LiDAR-based 3D object detection system that operates in real-time, with emphasis on autonomous driving scenarios. A state-of-the-art 2D standard object detector for RGB images, YOLOv4, is used as the base for object detection. The multi-class 3D bounding boxes are generated using a complex regression approach. An Euler-Region-Proposal Network (E-RPN) is used to predict the pose of the object. The proposed model receives point cloud data as input and outputs 3D bounding boxes with classes in real-time. The experiments done on the KITTI benchmark dataset proves that the proposed system outperforms existing methods in terms of accuracy and performance.

Autonomous driving emphasizes precise multi-sensor fusion positioning on limit resource embedded systems. LiDAR-centered sensor fusion system serves as a mainstream navigation system due to its insensitivity to illumination and viewpoint change. However, these types of systems suffer from handling large-scale sequential LiDAR data using limited resources on board, leading LiDAR-centralized sensor fusion unpractical. As a result, hand-crafted features such as plane and edge are leveraged in majority mainstream positioning methods to alleviate this unsatisfaction, triggering a new cornerstone in LiDAR Inertial sensor fusion. However, such super light weight feature extraction, although it achieves real-time constraint in LiDAR-centered sensor fusion, encounters severe vulnerability under high speed rotational or translational perturbation. In this paper, we propose a sparse tensor based LiDAR Inertial fusion method for autonomous driving embedded system. Leveraging the power of sparse tensor, the global geometrical feature is fetched so that the point cloud sparsity defect is alleviated. Inertial sensor is deployed to conquer the time-consuming step caused by the coarse level point-wise inlier matching. We construct our experiments on both representative dataset benchmarks and realistic scenes. The evaluation results show the robustness and accuracy of our proposed solution compared to classical methods.

Typical autonomous robotics systems incorporate multiple cameras, LIDAR sensors and sophisticated computing resources. In this paper we present a software framework for utilizing any array of multiple cameras with sufficient field-of-view (FOV) overlap as a light field imaging system. We show that the typical linear arrays that exist on autonomous cars are sufficient to capture stable time resolved light fields even when moving at highway speeds. We elaborate on the potential pitfalls associated with such a technique namely loss of calibration between cameras due to high frequency vibrations and sudden shocks associated with driving over potholes and highlight a method that can compensate for such effects. We demonstrate that the light fields collected by simple linear arrays can be processed in real time for a wide variety of useful applications including occlusion removal, for signal enhancement in featureless images captured in very low light, for reflection removal and for improved visibility in extreme conditions associated with snow and heavy rain.

Autonomous cars will reduce the quantity of street injuries because of human mistakes. Traditionally, a step by step advancement has been introduced in intelligent vehicles. These advancements boom the automation stage in motors with systems that assist the driving force in order to keep a steady speed, comply with a lane, or perform vehicle– driver control handover. Autonomous cars require deep information about their surroundings. Thus it's very vital to carry out road segmentation and object detection for identifying the drivable area in route planning. A portion of road with uniform characteristics can be represented specifically as segments. Existing road segmentation algorithms basically depends on quality of image from camera due to light conditions. As well one more issue for existing road segmentation depends on high quality of lane markings. LiDAR sensors can attain the 3D geometry information of the vehicle surroundings with excessive precision. The information collected by LiDAR will increase precision at the cost of increased memory usage and hence computational overhead. A brand new method using both the Camera and LiDAR data fusion for road detection is discussed. It decreases the gap among the point cloud of 3-D LiDAR information.

Robust pedestrian detection has been treated as one of the main pursuits for excellent autonomous driving. Recently, some convolutional neural networks (CNN) based detectors have made large progress for this goal, such as Faster R-CNN. However, the performance of them still needs a large space to be boosted, even owning the complex learning architectures. In this paper, we novelly introduce the 3D LiDAR sensor to boost the CNN-based pedestrian detection. Facing the heterogeneous and asynchronous properties of two different sensors, we firstly introduce an accurate calibration method for visual and LiDAR sensors. Then, some physically geometrical clues acquired by 3D LiDAR are explored to eliminate the erroneous pedestrian proposals generated by the state-of-the-art CNN-based detectors. Exhaustive experiments verified the superiority of the proposed method.

Predicting the future trajectories of surrounding vehicles is a key competence for safe and efficient real-world autonomous driving systems. Previous works have presented deep neural network models for predictions using a detailed prior map which includes driving lanes and explicitly expresses the road rules like legal traffic directions and valid paths through intersections. Since it is unrealistic to assume the existence of the detailed prior maps for all areas, we use a map generated from only perceptual data (3D points measured by a LiDAR sensor). Such maps do not explicitly denote road rules, which makes prediction tasks more difficult. To overcome this problem, we propose a novel generative adversarial network (GAN) based framework. A discriminator in our framework can distinguish whether predicted trajectories follow road rules, and a generator can predict trajectories following it. Our framework implicitly extracts road rules by projecting trajectories onto the map via a differentiable function and training positional relations between trajectories and obstacles on the map. We also extend our framework to multimodal predictions so that various future trajectories are predicted. Experimental results show that our method outperforms other state-of-the-art methods in terms of trajectory errors and the ratio of trajectories that fall on drivable lanes.

Positron emission tomography (PET) is a highly sensitive medical imaging technique commonly used to detect and assess tumor lesions. Magnetic resonance imaging (MRI) provides high resolution anatomical images with different contrasts and a range of additional information important for cancer diagnosis. Recently, simultaneous PET-MR systems have been released with the promise to provide complementary information from both modalities in a single examination. Due to long scan times, subject nonrigid bulk motion, i.e., changes of the patient's position on the scanner table leading to nonrigid changes of the patient's anatomy, during data acquisition can negatively impair image quality and tracer uptake quantification. A 3D MR-acquisition scheme is proposed to detect and correct for nonrigid bulk motion in simultaneously acquired PET-MR data. A respiratory navigated three dimensional (3D) MR-acquisition with Radial Phase Encoding (RPE) is used to obtain T1- and T2-weighted data with an isotropic resolution of 1.5 mm. Healthy volunteers are asked to move the abdomen two to three times during data acquisition resulting in overall 19 movements at arbitrary time points. The acquisition scheme is used to retrospectively reconstruct dynamic 3D MR images with different temporal resolutions. Nonrigid bulk motion is detected and corrected in this image data. A simultaneous PET acquisition is simulated and the effect of motion correction is assessed on image quality and standardized uptake values (SUV) for lesions with different diameters. Six respiratory gated 3D data sets with T1- and T2-weighted contrast have been obtained in healthy volunteers. All bulk motion shifts have successfully been detected and motion fields describing the transformation between the different motion states could be obtained with an accuracy of 1.71 ± 0.29 mm. The PET simulation showed errors of up to 67% in measured SUV due to bulk motion which could be reduced to less than 10% with the proposed motion compensation approach. A MR acquisition scheme which yields both high resolution 3D anatomical data and highly accurate nonrigid motion information without an increase in scan time is presented. The proposed method leads to a strong improvement in both MR and PET image quality and ensures an accurate assessment of tracer uptake.

A comprehensive study on lane detecting autonomous car using computer vision

The increase in extreme weather events due to climate change has led to irregular patterns in the hydrological cycle. In East Asia, a region characterized by a monsoon climate, natural disasters such as droughts and floods have become increasingly prevalent. This trend has underscored the necessity for effective soil moisture monitoring, as it is a crucial element in the hydrological cycle. To this end, various machine learning techniques based on satellite data combined with in-situ soil moisture observations are being actively researched for precise soil moisture estimation. However, the existing satellite images have limitations in temporal resolution compared to in-situ observations, and there is a need to improve the temporal resolution. In this study, soil moisture was estimated by linear regression using Cyclone Global Navigation Satellite System (CYGNSS) reflectivity and Soil Moisture Active Passive Level 2 (SMAP L2) soil moisture, vegetation opacity, surface roughness, and soil surface temperature in East Asia. The CYGNSS-based soil moisture was validated alongside SMAP Level 4 (L4) and Advanced SCATterometer (ASCAT) L2 data using Extended Triple Collocation (ETC) analysis, which demonstrated the high accuracy of CYGNSS. The results of this study provide high temporal resolution soil moisture data for East Asia, which can contribute to efficient hydrological factor monitoring and management.This research was supported by the BK21 FOUR (Fostering Outstanding Universities for Research) funded by the Ministry of Education (MOE, Korea) and National Research Foundation of Korea (NRF). This work is financially supported by Korea Ministry of Land, Infrastructure and Transport (MOLIT) as「Innovative Talent Education Program for Smart City」. This work was supported by Korea Environment Industry & Technology Institute (KEITI) through Research and Development on the Technology for Securing the Water Resources Stability in Response to Future Change Project, funded by Korea Ministry of Environment (MOE)(RS-2024-00332300). This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-00416443). This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2022R1A2C2010266).

A behavioural gap detection paradigm was used to determine the temporal resolution for song patterns by female crickets, Gryllus bimaculatus. For stimuli with a modulation depth of 100% the critical gap duration was 6-8ms. A reduction of the modulation depth of gaps to 50% led either to an increase or a decrease of the critical gap duration. In the latter case, the critical gap duration dropped to 3-4ms indicating a higher sensitivity of auditory processing. The response curve for variation of pulse period was not limited by temporal resolution. However, the reduced response to stimuli with a high duty cycle, and thus short pause durations, was in accordance with the limits of temporal resolution. The critical duration of masking pulses inserted into pauses was 4-6ms. An analysis of the songs of males revealed that gaps (5.8ms) and masking pulses (6.9ms) were at detectable time scales for the auditory pathway of female crickets. However, most of the observed temporal variation of song patterns was tolerated by females. Critical cues such as pulse period and pulse duty cycle provided little basis for inter-individual selection by females.

Object tracking in a video sequence is usually achieved by tracking the bounding box over the object or the object’s boundary, each of which has somewhat different applications. In this paper, we present a new open-contour tracking algorithm based on a Bayesian framework in which the contour is a part of the object’s boundary. We first propose a new state-space model for the representation of contours, which can handle the rigid and nonrigid motions of contours independently. This model enables us to focus on the nonrigid motions during the training, and the model works for challenging rigid motion scenarios. In addition, for the robust tracking of contours, we propose a measurement function that considers the contrast on object boundaries, target appearance, and temporal coherence. We applied the proposed method to two kinds of open-contours targets, and the experimental results show that the proposed method achieves superior performance to the conventional contour tracking methods. The proposed method is also compared with recent bounding box tracking methods for the object tracking purposes, and the comparison shows that the proposed method works robustly to fast motions and yields a more accurate estimate of an object’s location than the conventional bounding box tracking methods.

As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution. It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams. Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed. In this tutorial-like article, a comprehensive review of the emerging technology is given. First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced. The signal processing techniques for event noise processing and event data representation are then discussed. Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed. Finally, challenges and future research directions are pointed out. It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities.