Valeo (Ireland)

Abstract Depth estimation from 2D images is an essential task in computer vision with applications in scene understanding, robotics, and autonomous systems. The performance of supervised depth models depends on network design, loss formulation, data quality, and fine-tuning strategy. In this study, we propose a progressive fine-tuning approach for metric (absolute-scale) depth estimation. Our method uses transfer learning across multiple indoor datasets: real, synthetic, and pseudo-labelled. DenseNet-169 and EfficientNet-B0 backbones are fine-tuned on MIT-G, SUN-RGBD, SceneNet, and NYU2. We apply a three-scale combined loss with weighted MAE + Edge + SSIM terms at full, 1/2, and 1/4 resolution, and add a perceptual VGG component, while we keep the global coefficients of the loss at 1 for simplicity and reproducibility. We find that EfficientNet performs better on the smaller datasets, while DenseNet benefits most from the million-image SceneNet stage and reaches REL 0.105 and RMSE 0.359 on NYU2, comparable to recent transformer baselines yet using 6 $$\times $$ × fewer parameters. The pseudo-labelled MIT-G data is used as a warm-start and shows the potential of reducing annotation cost. All headline metrics results are based on sensor ground-truth data, avoiding circular evaluation. Qualitative analysis and zero-shot tests on the unseen iBims-1 benchmark confirm that the models generalise and produce coherent, detailed depth maps across diverse indoor scenes. The proposed pipeline thus offers a balanced trade-off between accuracy and computational cost for practical indoor depth estimation.

Event-based vision sensors have a higher temporal resolution, a wider dynamic range, and a lower latency than conventional frame-based cameras. For these reasons, event-based sensors are being considered for advanced driver assistance systems applications. If these sensors are to be used for automotive sensing and perception, their performance under adverse conditions, such as rain and fog, must be characterized to ensure reliable performance. This study presents a suite of tests using an event-based sensor under controlled conditions across a range of rainfall rates, ambient light levels, fog visibility levels and distances from the targets. To evaluate the performance of these sensors, the average event rate (number of events per second) was compared with rainfall rates and visibility. The results indicated that the diameter of the raindrops had a larger effect on the number of events than the rainfall rate. Furthermore, the investigation revealed that, by carefully configuring the camera settings, it is possible to mitigate the effects of rain on the sensor output.

As automated vehicles progress toward increasing levels of autonomy, the need for thorough testing of such systems in all relevant environments increases. These safety-critical systems often rely on visible-spectrum cameras to perceive the environment. Therefore, these systems must perform reliably under a range of adverse weather conditions. This study investigates the impact of rain on the quality of images taken in an experimental setting designed to vary rain intensity in a controlled manner. This study analyzes the impact of rain using low-level metrics such as contrast and spatial frequency response. In addition, overall image quality was evaluated using a range of full-reference image quality metrics. The results show a 45% reduction in SNR at 40 m and 38 mm/h, a 70% maximum decrease in Weber contrast at 30 m and 38 mm/h, and a 42% increase in color error as a result of rain in the environment. Consequently, degradation in image quality is likely to affect subsequent downstream computer vision performance. The results of this study highlight the need for robust testing and optimization of camera systems.

Hyperspectral Imaging (HSI), known for its advantages over traditional RGB imaging in remote sensing, agriculture, and medicine, has recently gained attention for enhancing Advanced Driving Assistance Systems (ADAS) perception. A few HSI datasets such as HyKo, HSI-Drive, HSI-Road, and Hyperspectral City have been made available. However, a comprehensive evaluation of semantic segmentation models (SSM) using these datasets is lacking. To address this gap, we evaluated the available annotated HSI datasets on four deep learning-based baseline SSMs i.e. DeepLab v3+, HRNet, PSPNet, and U-Net along with its two variants: Coordinate Attention (UNet-CA) and Convolutional Block-Attention Module (UNet-CBAM). The original models' architectures were adapted to handle the varying spatial and spectral dimensions of the datasets. These baseline SSMs were trained using class-weighted loss function for individual HSI datasets and were evaluated over mean-based metrics i.e. intersection over union (IoU), recall, precision, F1 score, specificity, and accuracy. Our results indicate that UNet-CBAM which extracts channel-wise feature extraction, outperforms other SSMs and shows the potential to leverage spectral information for enhanced semantic segmentation. This study establishes a baseline SSM-based benchmark on available annotated datasets for future evaluation of HSI-based ADAS perception. However, the limitations of current HSI datasets, such as limited dataset size, high class imbalance, and lack of fine-grained annotations, remain significant constraints for developing robust SSMs for ADAS applications.

Hierarchical topological representations can significantly reduce search times within mapping and localization algorithms. Although recent research has shown the potential for such approaches, limited consideration has been given to the suitability and comparative performance of different global feature representations within this context. In this work, we evaluate state-of-the-art hand-crafted and learned global descriptors using a hierarchical topological mapping technique on benchmark datasets and present results of a comprehensive evaluation of the impact of the global descriptor used. Although learned descriptors have been incorporated into place recognition methods to improve retrieval accuracy and enhance overall recall, the problem of scalability and efficiency when applied to longer trajectories has not been adequately addressed in a majority of research studies. Based on our empirical analysis of multiple runs, we identify that continuity and distinctiveness are crucial characteristics for an optimal global descriptor that enable efficient and scalable hierarchical mapping, and present a methodology for quantifying and contrasting these characteristics across different global descriptors. Our study demonstrates that the use of global descriptors based on an unsupervised learned Variational Autoencoder (VAE) excels in these characteristics and achieves significantly lower runtime. It runs on a consumer grade desktop, up to 2.3x faster than the second best global descriptor, NetVLAD, and up to 9.5x faster than the hand-crafted descriptor, PHOG, on the longest track evaluated (St Lucia, 17.6 km), without sacrificing overall recall performance.

Automated vehicles rely heavily on image data from visible spectrum cameras to perform a wide range of tasks from object detection, classification, and avoidance to path planning. The availability and reliability of these sensors in adverse weather is therefore of critical importance to the safe and continuous operation of an automated vehicle. This review paper presents a data communication-inspired Image Formation Framework that characterizes the data flow from object through channel to sensor, and subsequent processing of the data. This framework is used to explore the degree to which adverse weather conditions impact the cameras used on automated vehicles for sensing and perception. The effect of rain on each element of the model is reviewed. Furthermore, the prevalence of the appearance of these rain-induced changes in publicly available, open-source datasets is reviewed. The degree to which synthetic rain generation techniques can accurately capture these changes is also examined. Finally, the paper offers some suggestions on how future adverse weather automotive datasets should be collected.

Cameras are the primary sensor in automated driving systems. They provide high information density and are optimal for detecting road infrastructure cues laid out for human vision. Surround-view camera systems typically comprise of four fisheye cameras with 190°+ field of view covering the entire 360° around the vehicle focused on near-field sensing. They are the principal sensors for low-speed, high accuracy, and close-range sensing applications, such as automated parking, traffic jam assistance, and low-speed emergency braking. In this work, we provide a detailed survey of such vision systems, setting up the survey in the context of an architecture that can be decomposed into four modular components namely Recognition, Reconstruction, Relocalization, and Reorganization. We jointly call this the 4R Architecture. We discuss how each component accomplishes a specific aspect and provide a positional argument that they can be synergized to form a complete perception system for low-speed automation. We support this argument by presenting results from previous works and by presenting architecture proposals for such a system. Qualitative results are presented in the video at https://youtu.be/ae8bCOF77uY.

Moving object detection and segmentation is an essential task in the Autonomous Driving pipeline. Detecting and isolating static and moving components of a vehicle's surroundings are particularly crucial in path planning and localization tasks. This paper proposes a novel real-time architecture for motion segmentation of Light Detection and Ranging (LiDAR) data. We use three successive scans of LiDAR data in 2D Bird's Eye View (BEV) representation to perform pixel-wise classification as static or moving. Furthermore, we propose a novel data augmentation technique to reduce the significant class imbalance between static and moving objects. We achieve this by artificially synthesizing moving objects by cutting and pasting static vehicles. We demonstrate a low latency of 8 ms on a commonly used automotive embedded platform, namely Nvidia Jetson Xavier. To the best of our knowledge, this is the first work directly performing motion segmentation in LiDAR BEV space. We provide quantitative results on the challenging SemanticKITTI dataset, and qualitative results are provided in https://youtu.be/2aJ-cL8b0LI.

3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suitable for efficient parallelization. Secondly, they have a variable runtime due to level of sparsity of the scene which is against the determinism needed in a safety system. In this work, we aim to develop a very low latency algorithm with fixed runtime. We propose a novel semantic segmentation architecture as a single unified model for object center detection using key points, box predictions and orientation prediction using binned classification in a simpler Bird's Eye View (BEV) 2D representation. The proposed architecture can be trivially extended to include semantic segmentation classes like road without any additional computation. The proposed model has a latency of 4 ms on the embedded Nvidia Xavier platform. The model is 5X faster than other top accuracy models with a minimal accuracy degradation of 2% in Average Precision at IoU=0.5 on KITTI dataset.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.