A depth estimation method for edge precision improvement of depth map

In this study, we present a method for all-around depth estimation from multiple omnidirectional images for indoor environments. In particular, we focus on plane-sweeping stereo as the method for depth estimation from the images. We propose a new icosahedron-based representation and ConvNets for omnidirectional images, which we name "CrownConv" because the representation resembles a crown made of origami. CrownConv can be applied to both fisheye images and equirectangular images to extract features. Furthermore, we propose icosahedron-based spherical sweeping for generating the cost volume on an icosahedron from the extracted features. The cost volume is regularized using the three-dimensional CrownConv, and the final depth is obtained by depth regression from the cost volume. Our proposed method is robust to camera alignments by using the extrinsic camera parameters; therefore, it can achieve precise depth estimation even when the camera alignment differs from that in the training dataset. We evaluate the proposed model on synthetic datasets and demonstrate its effectiveness. As our proposed method is computationally efficient, the depth is estimated from four fisheye images in less than a second using a laptop with a GPU. Therefore, it is suitable for real-world robotics applications. Our source code is available at https://github.com/matsuren/crownconv360depth.

Quality assessment of stereoscopic video in free viewpoint video system

We have been developing a new television system named Free Viewpoint Television (FTV) that can generate free views. We propose a new user interface of FTV using an automultiscopic display (multi-viewing autostereoscopic display) and a head tracking system. We made a head tracking system which used cameras and didn't need to attach any sensors to a person. We succeeded to extend the viewing zone of an automultiscopic display. It brought us to interface with FTV naturally.

An overview of 3D and free viewpoint video is given in this paper with special focus on related standardization activities in MPEG. Free viewpoint video allows the user to freely navigate within real world visual scenes, as known from virtual worlds in computer graphics. Suitable 3D scene representation formats are classified and the processing chain is explained. Examples are shown for image-based and model-based free viewpoint video systems, highlighting standards conform realization using MPEG-4. Then the principles of 3D video are introduced providing the user with a 3D depth impression of the observed scene. Example systems are described again focusing on their realization based on MPEG-4. Finally multi-view video coding is described as a key component for 3D and free viewpoint video systems. MPEG is currently working on a new standard for multi-view video coding. The conclusion is that the necessary technology including standard media formats for 3D and free viewpoint is available or will be available in the near future, and that there is a clear demand from industry and user side for such applications. 3DTV at home and free viewpoint video on DVD will be available soon, and will create huge new markets.

Depth No-Synthesis-Error Time-Consistent Preprocessing for Depth Coding in FVV System

In this paper we present a novel method for depth estimation from stereoscopic 360-degree video. As for now, no such methods have been presented in the literature. First we show mathematical foundations for correspondence matching polar coordinates. Then, the attained formulas are simplified to show relationship to depth estimation in rectified pairs of rectangular images. The proposed method has been implemented in Depth Estimation Reference Software (DERS) developed by Motion Picture Experts Group (MPEG) of International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Finally, we present experimental results in form of depth maps estimated for “Dancing 360” stereoscopic omnidirectional sequence.

Source depth estimation is always a problem in underwater acoustic area, because depth estimation is a nonlinear problem. Traditional depth estimation methods use a vertical line array, which has disadvantage of poor mobility due to the size of sensor array. In order to estimate source depth with a horizontal line array, we propose a matched-mode depth estimation method based on autoregressive (AR) wavenumber estimation for a moving source in shallow water waveguides. First, we estimate the mode wavenumbers using the improved AR modal wavenumber spectrum. Second, according to the mode wavenumber estimation, we estimate the mode amplitudes by the wavenumber spectrum, which is obtained by generalized Hankel transform. Finally, we estimate source depth estimation by the peak of source depth function wherein the data mode best matches the replica mode that is calculated using a propagation model. Compared with synthetic aperture beamforming, the proposed method exhibits a better performance in source depth estimation under low signal-to-noise ratio or the small range span. The robustness of the proposed method is illustrated by simulating the performance in mismatched environment.

Depth estimation is a crucial step for 2D/3D conversion from monoscopic video. In this paper, a novel method for depth estimation from motion with camera motion is proposed. In the proposed method, image matching using critical point filters is applied to extract the pixel-level motion field for each frame. As camera motion can bring pseudo motion vectors by image matching, and thus leading to depth ambiguity. To solve this problem, we propose to estimate the camera moving model using robust RANSAC algorithm. Then, the initial depth map is estimated by using the motion vectors without camera motion. Finally, the depth values of the pixels at the edges of moving objects are refined using a post filter based on homogeneous points. Experimental results show that the proposed method achieves considerable performances on depth map in presence of camera motion.

Self-supervised depth estimation methods can achieve competitive performance using only unlabeled monocular videos, but they suffer from the uncertainty of jointly learning depth and pose without any ground truths of both tasks. Supervised framework provides robust and superior performance but is limited by the scope of the labeled data. In this paper, we introduce SENSE, a novel learning paradigm for self-supervised monocular depth estimation that progressively evolves the prediction result using supervised learning, but without requiring labeled data. The key contribution of our approach stems from the novel use of the pseudo labels - the noisy depth estimation from the self-supervised methods. We surprisingly find that a fully supervised depth estimation network trained using the pseudo labels can produce even better results than its "ground truth". To push the envelope further, we then evolve the self-supervised backbone by replacing its depth estimation branch with that fully supervised network. Based on this idea, we devise a comprehensive training pipeline that alternatively enhances the two key branches (depth and pose estimation) of the self-supervised backbone network. Our proposed approach can effectively ease the difficulty of multi-task training in self-supervised depth estimation. Experimental results have shown that our proposed approach achieves state-of-the-art results on the KITTI dataset.

Assessment of depth to magnetic sources using high resolution aeromagnetic data of some parts of the Lower Benue Trough and adjoining areas, Southeast Nigeria

Predicting depth maps from monocular images has made an impressive performance in the past years. However, most depth estimation methods are trained with paired image-depth map data or multi-view images (e.g., stereo pair and monocular sequence), which suffer from expensive annotation costs and poor transferability. Although unsupervised domain adaptation methods are introduced to mitigate the reliance on annotated data, rare works focus on the unsupervised cross-scenario indoor monocular depth estimation. In this article, we propose to study the generalization of depth estimation models across different indoor scenarios in an adversarial-based domain adaptation paradigm. Concretely, a domain discriminator is designed for discriminating the representation from source and target domains, while the feature extractor aims to confuse the domain discriminator by capturing domain-invariant features. Further, we reconstruct depth maps from latent representations with the supervision of labeled source data. As a result, the feature extractor learned features possess the merit of both domain-invariant and low source risk, and the depth estimator can deal with the domain shift between source and target domains. We conduct the cross-scenario and cross-dataset experiments on the ScanNet and NYU-Depth-v2 datasets to verify the effectiveness of our method and achieve impressive performance.

Multi-view 4D reconstruction of human action has a number of applications in entertainment. This chapter describes a selection of application areas that are of interest to the broadcast, movie and gaming industries. In particular, free-viewpoint video techniques for special effects and sport post-match analysis are discussed. The appearance of human action is captured as 4D data represented by 3D volumetric or surface data over time. A review of recent approaches identifies two major classes: 4D reconstruction and model-based tracking. The second part of the chapter describes aspects of a practical implementation of a 4D reconstruction pipeline. Implementations of the popular visual hull are discussed, as a building block in many free-viewpoint video systems.

As the high voltage circuit breaker works, the metal parts are prone to many types of mechanical defects, such as plastic deformation, metal loss, corrosion, and so on. The defects will inevitably decrease the performance, mechanical reliability, lifetime, and other critical parameters of the breaker. Therefore, the detection of mechanical defects is of great help to reduce the mechanical failure rate of circuit breakers. In the present paper, a novel method based on edge detection and deep learning is proposed to detect the mechanical defects of the high voltage circuit breaker. Firstly, the high resolution images of the circuit breaker are photographed, and the components in the images are segmented in the minimum range by the contour detection algorithm. After binarization and morphological processing of the segmented images, the edge is drawn with the improved edge detection algorithm. However, the components of the circuit breaker are complicated, which results that the edge features of the critical component are difficult to extract. Then, the specific contour of components is detected by the depth estimation and segmentation methods. Secondly, a deep learning platform base on Tensorflow is established, which is an optimization method based on Convolutional Neural Network. In order to enhance the feature extraction capacity, a convolution kernel is inputted into a sparse self-coding network to proceeds with the optimal pre-training. Based on the above method, the features of images are automatically learned, and each convolution carries for the equilibrium of image entropy of convolution kernel by introducing similarity constraint rule. Finally, 3 types of mechanical defects, including plastic deformation, metal loss, corrosion, images and non-defect images as input are exploited to validate the CNN with convolution kernel optimized. The experimental results show that the average accuracy of the approach is better than the traditional convolution neural network model and has good feature extraction ability and generalization ability. In conclusion, combined with the improved edge detection and deep learning algorithms, the detection of typical mechanical defects can be well performed. For the collected images of circuit breaker, so as to reduce the impact of noise on defect accuracy of mechanical, the proposed method provides a reference for the non-destructive mechanical defect detection of high voltage circuit breaker.

Underground mapping is of paramount importance at archaeological sites with natural or man‐made caves. Techniques of different complexity are available for underground surveying, from compass and tape to light detection and ranging (LiDAR). However, there are scenarios where it is impossible to use heavy and/or delicate instrumentation, or with some of the more advanced techniques, long‐distance and time‐consuming fieldwork would be required. This is the case of the study of the height of water at key points inside the aquifer located in the subsoil of the Roman city of Colonia Clunia Sulpicia to assess its relation with the evolution of the city. Although subsurface radiolocation, a technique originally intended for cave mapping, seems promising for this application, its accuracy in depth estimation is not sufficient for this purpose. Therefore, the main objectives of this work are to improve the accuracy of depth estimation using radiolocation, to obtain sound data to study changes in water supply in the early centuries of the city and to hypothesize possible causes and probable consequences. Then, this work analyses the sources of error affecting the radiolocation process and develops a new method experimentally validated to improve the depth estimation accuracy. As a result, the depth of key points at Clunia has been accurately measured by radiolocation, where LiDAR or direct measurements from the water level are not possible. Finally, based on the resultant data, a chronology of the city in relation to the aquifer, and vice versa, is outlined. The new radiolocation and calculation procedure proposed in this paper is an improved subsurface location technique that can lead to a significant innovation in archaeological prospection.

Translation of fluorescence imaging using molecularly targeted imaging agents for real-time assessment of surgical margins in the operating room requires a fast and reliable method to predict tumor depth from planar optical imaging. Here, we developed a dual-wavelength fluorescent molecular probe with distinct visible and near-infrared excitation and emission spectra for depth estimation in mice and a method to predict the optical properties of the imaging medium such that the technique is applicable to a range of medium types. Imaging was conducted at two wavelengths in a simulated blood vessel and an in vivo tumor model. Although the depth estimation method was insensitive to changes in the molecular probe concentration, it was responsive to the optical parameters of the medium. Results of the intra-tumor fluorescent probe injection showed that the average measured tumor sub-surface depths were 1.31 ± 0.442 mm, 1.07 ± 0.187 mm, and 1.42 ± 0.182 mm, and the average estimated sub-surface depths were 0.97 ± 0.308 mm, 1.11 ± 0.428 mm, 1.21 ± 0.492 mm, respectively. Intravenous injection of the molecular probe allowed for selective tumor accumulation, with measured tumor sub-surface depths of 1.28 ± 0.168 mm, and 1.50 ± 0.394 mm, and the estimated depths were 1.46 ± 0.314 mm, and 1.60 ± 0.409 mm, respectively. Expansion of our technique by using material optical properties and mouse skin optical parameters to estimate the sub-surface depth of a tumor demonstrated an agreement between measured and estimated depth within 0.38 mm and 0.63 mm for intra-tumor and intravenous dye injections, respectively. Our results demonstrate the feasibility of dual-wavelength imaging for determining the depth of blood vessels and characterizing the sub-surface depth of tumors in vivo.