Novel View Synthesis in Time and Space

Novel view synthesis is an important problem in computer vision and graphics. Over the years a large number of solutions have been put forward to solve the problem. However, the large-baseline novel view synthesis problem is far from being solved. Recent works have attempted to use Convolutional Neural Networks (CNNs) to solve view synthesis tasks. Due to the difficulty of learning scene geometry and interpreting camera motion, CNNs are often unable to generate realistic novel views. In this paper, we present a novel view synthesis approach based on stereo-vision and CNNs that decomposes the problem into two sub-tasks: view dependent geometry estimation and texture inpainting. Both tasks are structured prediction problems that could be effectively learned with CNNs. Experiments on the KITTI Odometry dataset show that our approach is more accurate and significantly faster than the current state-of-the-art. The code and supplementary material will be publicly available. Results could be found here this https URL

Novel view synthesis is a challenging problem in computer vision and robotics. Different from the existing works, which need the reference images or 3D models of the scene to generate images under novel views, we propose a novel paradigm to this problem. That is, we synthesize the novel view from only a 6-DoF camera pose directly. Although this setting is the most straightforward way, there are few works addressing it. While, our experiments demonstrate that, with a concise CNN, we could get a meaningful parametric model that could reconstruct the correct scenery images only from the 6-DoF pose. To this end, we propose a two-stage learning strategy, which consists of two consecutive CNNs: GenNet and RefineNet. GenNet generates a coarse image from a camera pose. RefineNet is a generative adversarial network that refines the coarse image. In this way, we decouple the geometric relationship between mapping and texture detail rendering. Extensive experiments conducted on the public datasets prove the effectiveness of our method. We believe this paradigm is of high research and application value and could be an important direction in novel view synthesis.

Graph and hypergraph matching are fundamental problems in computer vision, which have been receiving a steadily growing attention in the last two decades due to the recent modern algorithms and models that are both fast and accurate. Graph and hypergraph matching are successful in tackling many real-world tasks that require 2D and 3D feature matching and alignment. Matching is a general problem in vision and graphs have always been quintessential for modeling information that has both local and longer range connections. Therefore, graph matching will continue to influence computer vision and robotics research, crossing over many different mathematical and computational models, including Bayesian networks, conditional and Markov random fields, spectral graph models, and, more recently, deep neural networks. While graph matching is limited to using pairwise relationships, hypergraph matching permits the use of relationships between sets of features of any order. Consequently, it carries the promise to make matching more robust to changes in scale, deformations, and outliers. In this chapter, we present several key ideas, computational models, and optimization methods that introduced in the literature the possibility to learn graph and hypergraph matching in a semi-supervised or even completely unsupervised fashion. We start by describing two methods for solving graph and hypergraph matching based on spectral matching and integer projected fixed point methods, introduced in Chap. 1. Then, based on the same intuition (also discussed in Chap. 1) that correct assignments form a strong cluster of agreements, we follow up by introducing the main approach to learning as it applies to graph and hypergraph matching, as well as the more general Maximum A Posteriori (MAP) inference problem in probabilistic graphical models (MRFs and CRFs). In the last part of the chapter, we demonstrate experimentally the effectiveness of the algorithms presented and, whenever possible, provide extensive comparisons to other top published methods. Moreover, we show that our learning approach is general and can significantly boost the performance of different state-of-the-art matching algorithms in the literature.

Depth-image-based rendering (DIBR) oriented view synthesis has been widely employed in the current depth-based 3-D video systems by synthesizing a virtual view from an arbitrary viewpoint. However, holes may appear in the synthesized view due to disocclusion, thus significantly degrading the quality. Consequently, efforts have been made on developing effective and efficient hole-filling algorithms. Current hole-filling techniques generally extrapolate/interpolate the hole regions with the neighboring information based on an assumption that the texture pattern in the holes is similar to that of the neighboring background information. However, in many scenarios, especially of complex texture, the assumption may not hold. In other words, hole-filling techniques can only provide an estimation for a hole which may not be good enough or may even be erroneous considering a wide variety of complex scene of images. In this paper, we first examine the view interpolation with multiple reference views, demonstrating that the problem of emerging holes in a target virtual view can be greatly alleviated by making good use of other neighboring complementary views in addition to its two (commonly used) most neighboring primary views. The effects of using multiple views for view extrapolation in reducing holes are also investigated in this paper. In view of the 3D Video and ongoing free-viewpoint TV standardization, we propose a new view synthesis framework, which employs multiple views to synthesize output virtual views. Furthermore, a scheme of selective warping of complementary views is developed by efficiently locating a small number of useful pixels in the complementary views for hole reduction, to avoid full warping of additional complementary views thus lowering greatly the warping complexity. Experimental results show that the hole size based on two primary reference views may be reduced by up to about 70% with the help of two complementary reference views in the case of view interpolation, while the hole size based on one primary reference view may be reduced by about 27% with the help of one more complementary reference view in view extrapolation. Moreover, it is shown that by using one more pair of views in view interpolation and one more view in view extrapolation, 10% hole pixels may be reduced additionally.

Toward designing intelligent PDEs for computer vision: An optimal control approach

Colour constancy is a classical problem in computer vision. Although there are a number of colour constancy algorithms based on different assumptions, none of them can be considered as universal. How to select or combine these available methods for different natural image characteristics is an important problem. Recent studies have shown that the texture feature is an important factor to consider when selecting the best colour constancy algorithm for a certain image. In this paper, Weibull parameterisation is used to identify the texture characteristics of colour images. According to the texture similarity, the best colour constancy method (or best combination of methods) is selected out for a specific image. The experiments were carried out on a large data set and the results show that this new approach outperforms current state‐of‐the‐art single algorithms, as well as some combined algorithms.

The shape-from-shading (SfS) problem is a classic problem in computer vision. The task in SfS is to compute on the basis of the shading variation in a given 2-D image the 3-D depth of the depicted scene. The corresponding mathematical model eventually leads to a boundary value problem for a Hamilton-Jacobi equation. In this paper we evaluate and compare suitable numerical methods. We begin with a brief discussion of four state-of-the-art-approaches in this field. Then we give an extensive numerical comparison, thus evaluating recent improvements in this area. In the course of doing this, we introduce efficient variations of existing schemes. By this systematic investigation, we complement and extend previous works on the numerical side. The paper is finished by a conclusion.

Deep multi-scale feature mixture model for image super-resolution with multiple-focal-length degradation

Homography refers to a specific type of transformation that relates two images of the same planar surface taken from different perspectives. Recovering motion parameters from a homography matrix is a classic problem in computer vision. It is important to derive a fast and stable solution to homography decomposition, since it forms a critical component of many vision systems, e . g ., in Structure-from-Motion and visual localization. The current state-of-the-art solvers can be categorized into two types of methods, the numerical procedures based on singular value decomposition (SVD), and the closed-form solution. The SVD-based methods are stable but time-consuming, while the existing closed-form solution is faster but less stable. In this paper, we discuss the homography decomposition problem from a different viewpoint. In contrast to the existing methods which focus on the properties of the homography matrix, we propose a new method that uses three random point correspondences to obtain the motion parameters in closed form. The proposed method is conceptually simple, easy to understand and implement, and has a good geometrical interpretation. This solution can be seen as an alternative to the existing closed-form solution. We also discuss the configurations where the closed-form solutions might be unstable and present a framework for homography decomposition taking into account both the efficiency and stability.

Stereo matching is a classical problem in computer vision. It has been widely used in many fields, especially autonomous driving in recent years. Two key aspects of speed and accuracy are both desirable but conflicting characteristics in autonomous driving. In this paper, we present CMNet, a lightweight stereo matching architecture for improving the trade-off between speed and accuracy on resource-limited devices. A novel feature extraction network consisted of a patch embedding layer and a ConvMLP-mixer is proposed. The patch embedding layer enhances the receptive field and makes the feature vectors compact. The accuracy of the disparity map is increased by mixing the spatial information in the channel dimension through the ConvMLP-mixer. The absolute difference volume is concatenated with the group-wise correlation volume to provide multi-dimensional matching cost information for the cost aggregation stage. Being evaluated on KITTI 2012 and KITTI 2015 stereo matching datasets, the inference time of CMNet on NVIDIA GTX 2080ti GPU is 8.7 ms. While realizing fast predictions beyond real-time, the results of D1-all are 3.41% on KITTI 2012 and 3.84% on KITTI 2015, achieving state-of-the-art result between speed and accuracy. Besides, the lightweight architecture of CMNet enables a fast inference time of 40.7 ms on Nvidia Jetson Nano to realize real-time applications on edge devices.

The perspective-three-point problem (P3P) is a classical problem in computer vision. The existing direct solutions of P3P have at least three limitations: (1) the numerical instability when using different vertex permutations, (2) the degeneration in the geometric singularity case, and (3) the dependence on particular equation solvers. A new direct solution of P3P is presented to deal with these limitations. The main idea is to reduce the number of unknown parameters by using a geometric constraint we called "perspective similar triangle" (PST). The PST method achieves high stability in the permutation problem and in the presence of image noise, and does not rely on particular equation solvers. Furthermore, reliable results can be retrieved even in "danger cylinder", a typical kind of geometric singularity of P3P, where all existing direct solutions degenerate significantly.

The image super-resolution (SR) technique, which aims at reconstructing a high-resolution (HR) image from a single low-resolution (LR) image, is a classical problem in computer vision. Limited by the imaging hardware, the spatial resolution of a hyperspectral images (HSI) is usually very coarse. Meanwhile, the spectral information of the HSI is extremely important for its applications and cannot be severely distorted. This paper presents a spatial constraint (SCT) strategy with combination of a deep learning method for HSI SR. The SCT strategy restraints the LR HSI generated by the reconstructed HR HSI should be spatially close to the input LR HSI. The deep learning method learns an end-to-end mapping between the spectral difference of the LR HSI and that of the HR HSI. The mapping is represented as a deep convolutional neural network (CNN). The CNN learned spectral difference is utilized to super-resolve the LR HSI while preserve the important spectral information of the desired HR HSI. Experiments have been conducted on three databases that contains both indoor scenes and outdoor scenes. Comparative analyses have verified the effectiveness of the overall method.

The integration of surface normals for the purpose of computing the shape of a surface in 3D space is a classic problem in computer vision. However, even nowadays it is still a challenging task to devise a method that is flexible enough to work on non-trivial computational domains with high accuracy, robustness, and computational efficiency. By uniting a classic approach for surface normal integration with modern computational techniques, we construct a solver that fulfils these requirements. Building upon the Poisson integration model, we use an iterative Krylov subspace solver as a core step in tackling the task. While such a method can be very efficient, it may only show its full potential when combined with suitable numerical preconditioning and problem-specific initialisation. We perform a thorough numerical study in order to identify an appropriate preconditioner for this purpose. To provide suitable initialisation, we compute this initial state using a recently developed fast marching integrator. Detailed numerical experiments illustrate the benefits of this novel combination. In addition, we show on real-world photometric stereo datasets that the developed numerical framework is flexible enough to tackle modern computer vision applications.

3D reconstruction from a single image is a classical problem in computer vision. However, it still poses great challenges for the reconstruction of daily-use objects with irregular shapes. In this paper, we propose to learn 3D reconstruction knowledge from informally captured RGB-D images, which will probably be ubiquitously used in daily life. The learning of 3D reconstruction is defined as a category modeling problem, in which a model for each category is trained to encode category-specific knowledge for 3D reconstruction. The category model estimates the pixel-level 3D structure of an object from its 2D appearance, by taking into account considerable variations in rotation, 3D structure, and texture. Learning 3D reconstruction from ubiquitous RGB-D images creates a new set of challenges. Experimental results have demonstrated the effectiveness of the proposed approach.

Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clustering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves clustering performance in a training-free manner. In our approach, we incorporate two complementary types of textual semantics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we design a residual attention mechanism to further enhance the discriminability of this space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.