Unsupervised Monocular Depth Estimation Based on Residual Neural Network of Coarse–Refined Feature Extractions for Drone

Deployment of depth estimating models in computer vision applications like obstacle avoidance, scene reconstruction, and camera pose estimation has become a fundamental task in the present day. Most of the depth images generated by existing monocular depth estimation models have blurry approximations of the depth and resolution, especially in low-textured regions. However, the existing depth estimation models that provide better predictions of depth generally take multiple images of the same scene as an input for generating depth maps. This paper presents a transfer learning approach with densely connected convolutional neural networks that take only an RGB image as an input for deeper and high-quality prediction of depth. In the proposed solution, an encoder-decoder architecture is leveraged for extracting features from an RGB image and generating a depth map for that corresponding RGB image. The densely connected convolutional network with 161 layers (Densenet-161) is used as an encoder, and the decoder is made up of five upsampling blocks and one transposed convolutional layer. The evaluation results obtained after training the model on the benchmark NYU V2 depth dataset ( Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor Segmentation and Support Inference from RGBD Images. In: Proceedings of the IEEE Conference on Computer Vision, pp. 746-760(2012).) have shown that the proposed approach for monocular depth estimation with the encoder-decoder architecture of lesser complexity, trained with fewer parameters and iterations, performs better than the existing state-of-the-art techniques as the value of the average Root Mean Square Error (RMSE) between the predicted and ground-truth depths, calculated in the evaluation phase of the proposed approach, is 0.505 which is less than the RMSE values obtained in case of all the mentioned monocular depth estimation methods. Furthermore, the depth maps generated by the proposed model have a good quality resolution and have minimal effect on the surrounding conditions like the texture of the walls, illumination effects, etc.

This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Moreover, the Transformer and convolution are good at long-range and close-range depth estimation, respectively. Therefore, we propose to adopt a parallel encoder architecture consisting of a Transformer branch and a convolution branch. The former can model global context with the effective attention mechanism and the latter aims to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features and model the affinity between the heterogeneous features in a set-to-set translation manner. Due to the unbearable memory cost introduced by the global attention on high-resolution feature maps, we adopt the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. The effectiveness of each proposed module is elaborately evaluated through meticulous and intensive ablation studies.

Monocular depth estimation with guidance of surface normal map

Camouflaged object detection (COD) faces unique challenges due to the extremely high visual similarity between objects and their surroundings, coupled with indistinct boundary features. While the introduction of depth information has provided new insights into addressing these challenges, existing methods still exhibit considerable limitations in depth data quality assessment and optimization. To address this issue, this paper proposes a depth screening and calibration (DSC) framework aimed at constructing a high-quality RGBD COD dataset. The framework first establishes a comprehensive evaluation metric that quantitatively assesses depth data generated by various monocular depth estimation (MDE) methods across multiple dimensions, including structural similarity, edge consistency, foreground smoothness, depth value utilization, and depth disparity between foreground and background. Based on these metrics, optimal depth maps are selected from those generated by multiple MDE methods for each image, forming an initial RGBD COD dataset. Subsequently, a Two-stage Depth Calibration (TDC) strategy is designed to calibrate the depth maps in the initial dataset through two consecutive phases: positive-negative sample discrimination and calibrated depth map generation, effectively enhancing the overall quality of depth maps. Experimental results on three benchmark datasets demonstrate that detection models trained with our high-quality depth data significantly outperform alternative approaches. This work provides a reliable data foundation for further exploring the role of depth information in improving COD performance.

In recent years, self-supervised methods for monocular depth estimation has rapidly become an significant branch of depth estimation task, especially for autonomous driving applications. Despite the high overall precision achieved, current methods still suffer from a) imprecise object-level depth inference and b) uncertain scale factor. The former problem would cause texture copy or provide inaccurate object boundary, and the latter would require current methods to have an additional sensor like LiDAR to provide depth ground-truth or stereo camera as additional training inputs, which makes them difficult to implement. In this work, we propose to address these two problems together by introducing DNet. Our contributions are twofold: a) a novel dense connected prediction (DCP) layer is proposed to provide better object-level depth estimation and b) specifically for autonomous driving scenarios, dense geometrical constrains (DGC) is introduced so that precise scale factor can be recovered without additional cost for autonomous vehicles. Extensive experiments have been conducted and, both DCP layer and DGC module are proved to be effectively solving the aforementioned problems respectively. Thanks to DCP layer, object boundary can now be better distinguished in the depth map and the depth is more continues on object level. It is also demonstrated that the performance of using DGC to perform scale recovery is comparable to that using ground-truth information, when the camera height is given and the ground point takes up more than 1.03% of the pixels. Code is available at https://github.com/TJ-IPLab/DNet.

Unsupervised monocular depth estimation is challenging in ill-posed regions, such as weak texture scenes, projection occlusion, and redundant error of detail information, etc. In this paper, in order to tackle these problems, an improved unsupervised monocular depth estimation method for the ill-posed region is proposed through cascading training depth estimation network and pose estimation network by loss function. Firstly, for the depth estimation network, a feature extraction network using asymmetric convolution is designed instead of traditional convolution, which strengthens the extraction of the feature information and improves the accuracy of the weak texture scenes. Meanwhile, a feature extraction network integrating multi-scale receptive fields with the structure of different scale convolution and dilated convolution stack is designed to increase the underlying receptive field of the depth estimation network, which strengthens the fusion ability of the network for multi-scale detail information, and improves the integrity of the model output details. Secondly, a pose estimation network using an attention mechanism is presented to strengthen the pose detail information of keyframes and suppress redundant errors of the pose information of non-keyframes. Finally, a loss function with minimum reprojection error is adopted to alleviate the occlusion problem of the projection process between adjacent pixels and enhance the quality of the output depth images of the model. The experiments demonstrate that our method achieves state-of-the-art performance on KITTI monocular datasets.

SABV-Depth: A biologically inspired deep learning network for monocular depth estimation

Monocular depth estimation infers 3D geometric structures of scenes from a single RGB image, offering significant applications in autonomous driving, robot navigation, and other fields. While current self-supervised learning methods avoid dependency on ground truth depth data, they still exhibit no-table limitations in complex scenarios: traditional encoder-decoder architec-tures inevitably lose high-frequency detail features when acquiring global context through continuous downsampling, resulting in blurred edges and texture distortion in depth maps. To alleviate these issues, we propose a nov-el approach named HyperDetailNet, which significantly enhances depth es-timation detail preservation. Specifically, our method contains two key com-ponents: 1 A dual-branch detail-global feature extraction network, where the detail branch adopts an enlarge-then-reduce strategy to preserve high-frequency texture information, while the global branch extracts overall struc-tural information of the scene. 2 To effectively fuse features from both branches, we designed a multi-attention fusion module that combines spatial attention, channel attention, and sliding window self-attention mechanisms to enhance model perception of detailed regions. Experimental results demonstrate that HyperDetailNet achieves excellent performance on both KITTI and Make3D datasets, with significant improvements in depth estima-tion for edge and texture-rich areas. Additionally, ablation experiments veri-fy the effectiveness of the dual-branch detail-global feature extraction DepthNet and multi-attention fusion module.

Depth estimation is an essential component in understanding the 3D geometry of a scene. In comparison to traditional depth estimation methods such as structure from motion and stereo vision matching, determining depth relation using a single camera is challenging. The recent advancements in convolutional neural networks have accelerated the research in monocular depth estimation. However, most technologies infer depth maps using lower resolution images due to network capacity and complexity issues. Another challenge in depth estimation is ambiguous and sparse depth maps. These issues are caused due to labeling errors, hardware faults, or occlusions. This paper presents a novel end-to-end trainable convolutional neural network architecture – depth transverse transformer network (DTTNet). The proposed network is designed and optimized to perform monocular depth estimation. This network aims at exploring the multi-resolution representation to perform pixel-wise depth estimation more accurately. In order to improve the accuracy of depth estimation, different kinds of ad hoc networks are proposed subsequently. Extensive computer simulations on NYU Depth V2 and SUN RGB-D dataset demonstrate the effectiveness of the proposed DTTNet against state-of-the-art methods. DTTNet can potentially optimize depth perception in intelligent systems such as automated driving and video surveillance applications, computational photography, and augmented reality. The source code is available at https://github.com/shreyaskamathkm/DTTNet

Monocular depth estimation is a fundamental yet challenging task in computer vision as depth information will be lost when 3D scenes are mapped to 2D images. Although deep learning-based methods have led to considerable improvements for this task in a single image, most existing approaches still fail to overcome this limitation. Supervised learning methods model depth estimation as a regression problem and, as a result, require large amounts of ground truth depth data for training in actual scenarios. Unsupervised learning methods treat depth estimation as the synthesis of a new disparity map, which means that rectified stereo image pairs need to be used as the training dataset. Aiming to solve such problem, we present an encoder-decoder based framework, which infers depth maps from monocular video snippets in an unsupervised manner. First, we design an unsupervised learning scheme for the monocular depth estimation task based on the basic principles of structure from motion (SfM) and it only uses adjacent video clips rather than paired training data as supervision. Second, our method predicts two confidence masks to improve the robustness of the depth estimation model to avoid the occlusion problem. Finally, we leverage the largest scale and minimum depth loss instead of the multiscale and average loss to improve the accuracy of depth estimation. The experimental results on the benchmark KITTI dataset for depth estimation show that our method outperforms competing unsupervised methods.

Monocular Depth Estimation (MDE) plays a vital role in applications such as autonomous driving. However, various attacks target MDE models, with physical attacks posing significant threats to system security. Traditional adversarial training methods, which require ground-truth labels, are not directly applicable to MDE models that lack ground-truth depth. Some self-supervised model hardening techniques (e.g., contrastive learning) overlook the domain knowledge of MDE, resulting in suboptimal performance. In this work, we introduce a novel self-supervised adversarial training approach for MDE models, leveraging view synthesis without the need for ground-truth depth. We enhance adversarial robustness against real-world attacks by incorporating L0-norm-bounded perturbation during training. We evaluate our method against supervised learning-based and contrastive learning-based approaches specifically designed for MDE. Our experiments with two representative MDE networks demonstrate improved robustness against various adversarial attacks, with minimal impact on benign performance.

Collaborative surgical instrument segmentation for monocular depth estimation in minimally invasive surgery.

Monocular depth estimation is vital for scene understanding and downstream tasks. We focus on the supervised setup, in which ground-truth depth is available only at training time. Based on knowledge about the high regularity of real 3D scenes, we propose a method that learns to selectively leverage information from coplanar pixels to improve the predicted depth. In particular, we introduce a piecewise planarity prior which states that for each pixel, there is a seed pixel which shares the same planar 3D surface with the former. Motivated by this prior, we design a network with two heads. The first head outputs pixel-level plane coefficients, while the second one outputs a dense off-set vector field that identifies the positions of seed pixels. The plane coefficients of seed pixels are then used to predict depth at each position. The resulting prediction is adaptively fused with the initial prediction from the first head via a learned confidence to account for potential deviations from precise local planarity. The entire architecture is trained end-to-end thanks to the differentiability of the proposed modules and it learns to predict regular depth maps, with sharp edges at occlusion boundaries. An extensive evaluation of our method shows that we set the new state of the art in supervised monocular depth estimation, surpassing prior methods on NYU Depth-v2 and on the Garg split of KITTI. Our method delivers depth maps that yield plausible 3D reconstructions of the input scenes. Code is available at: https://github.com/SysCV/P3Depth

In this paper, An advanced pothole detection method is proposed in this project with accurate calculation of the dimensions of the potholes, estimation of their volume, and determination of the filling cost of these potholes by using video inputs from road surfaces. Our system uses model object detection models to process every video frame to precisely identify potholes, and then me the dimensions as concerning length, breadth, and depth by using depth estimation model. Specifically, the system applies a technology known as monocular depth estimation, which allows for the estimation of the precise depth of every pothole directly from the feed obtained from one camera, enabling the calculation of volume to be repaired. Calculating the volume of each pothole, it computes the cavity volume and uses this volume to estimate the repair cost. Thus, detection, measurement, and cost estimation are fully automated in this solution. Consequently, the road inspection process becomes more streamlined and less laborious as compared to labor-intensive traditional inspection methods. This method is more accurate for pothole detection while it allows for assessment and planning for such extensive infrastructure maintenance works. . Keywords: Pothole detection, depth estimation, Volume calculation, Repair cost estimation, Cost-effective solution

As it is well known, some versions of the Pepper robot provide poor depth perception due to the lenses it has in front of the tridimensional sensor. In this paper, we present a method to improving that faulty 3D perception. Our proposal is based on a combination of the actual depth readings of Pepper and a deep learning-based monocular depth estimation. As shown, the combination of both of them provides a better 3D representation of the scene. In previous works we made an initial approximation of this fusion technique, but it had some drawbacks. In this paper we analyze the pros and cons of the Pepper readings, the monocular depth estimation method and our previous fusion method. Finally, we demonstrate that the proposed fusion method outperforms them all.