Shape from Shading

Shape-from-Shading (SfS) is a fundamental problem in Computer Vision. A very common assumption in this field is that image projection is orthographic. This paper re-examines the basis of SfS, the image irradiance equation, under a perspective projection assumption. The resultant equation does not depend on the depth function directly, but rather, on its natural logarithm. As such, it is invariant to scale changes of the depth function. A reconstruction method based on the perspective formula is then suggested; it is a modification of the Fast Marching method of Kimmel and Sethian. Following that, a comparison of the orthographic Fast Marching, perspective Fast Marching and the perspective algorithm of Prados and Faugeras on synthetic images is presented. The two perspective methods show better reconstruction results than the orthographic. The algorithm of Prados and Faugeras equates with the perspective Fast Marching. Following that, a comparison of the orthographic and perspective versions of the Fast Marching method on endoscopic images is introduced. The perspective algorithm outperformed the orthographic one. These findings suggest that the more realistic set of assumptions of perspective SfS improves reconstruction significantly with respect to orthographic SfS. The findings also provide evidence that perspective SfS can be used for real-life applications in fields such as endoscopy.

Shape-from-shading (SfS) is a fundamental problem in computer vision; it is based upon the image irradiance equation. Recently, the authors proposed to solve the image irradiance equation under the assumption of perspective projection rather than the common orthographic one. The solution was a modification of the fast marching method of Kimmel and Sethian. This paper presents an application of this novel perspective algorithm to reconstruction of medical images. We focus on gastrointestinal endoscopy and compare the two versions of the fast marching method (orthographic vs. perspective). The examples and comparison show that, unlike orthographic SfS, perspective SfS is robust and can be utilized for real-life applications.

Graph and hypergraph matching are fundamental problems in computer vision, which have been receiving a steadily growing attention in the last two decades due to the recent modern algorithms and models that are both fast and accurate. Graph and hypergraph matching are successful in tackling many real-world tasks that require 2D and 3D feature matching and alignment. Matching is a general problem in vision and graphs have always been quintessential for modeling information that has both local and longer range connections. Therefore, graph matching will continue to influence computer vision and robotics research, crossing over many different mathematical and computational models, including Bayesian networks, conditional and Markov random fields, spectral graph models, and, more recently, deep neural networks. While graph matching is limited to using pairwise relationships, hypergraph matching permits the use of relationships between sets of features of any order. Consequently, it carries the promise to make matching more robust to changes in scale, deformations, and outliers. In this chapter, we present several key ideas, computational models, and optimization methods that introduced in the literature the possibility to learn graph and hypergraph matching in a semi-supervised or even completely unsupervised fashion. We start by describing two methods for solving graph and hypergraph matching based on spectral matching and integer projected fixed point methods, introduced in Chap. 1. Then, based on the same intuition (also discussed in Chap. 1) that correct assignments form a strong cluster of agreements, we follow up by introducing the main approach to learning as it applies to graph and hypergraph matching, as well as the more general Maximum A Posteriori (MAP) inference problem in probabilistic graphical models (MRFs and CRFs). In the last part of the chapter, we demonstrate experimentally the effectiveness of the algorithms presented and, whenever possible, provide extensive comparisons to other top published methods. Moreover, we show that our learning approach is general and can significantly boost the performance of different state-of-the-art matching algorithms in the literature.

A fast marching algorithm for the factored eikonal equation

Finding epipolar geometry for two images is a fundamental problem in computer vision. While this typically relies on feature point correspondence, the epipolar constraint can also be used for improving the accuracy of correspondence. We propose a probabilistic framework for estimating the epiploar geometry, in which the geometry and the feature correspondence are estimated iteratively at the same time. Using the EM algorithm to maximize a posteriori, our approach updates feature correspondence with estimated epipolar geometry. The correspondence is further improved with local diffusion on a prior Markov Random Field model. In turn, more accurate epipolar geometry is recovered. Experiments show this approach produces more accurate fundamental matrix compared with typical methods and can handle some challenging situations such as view rotation and scale changes.

In the areas of computer vision and computer graphics, analyzing information about three-dimensional objects from two-dimensional images is essential. Shape from shading is a very fundamental problem in computer vision, which uses features and patterns in images to infer the shape of the object in our viewing direction. The influence of solving this problem is significant. For example, a typical application of shape from shading is in astronomy, in which we can obtain landscape features of a planet by analyzing the information from the image we get.

Finding the surface of a volumetric 3D object is a fundamental problem in computer vision. Energy minimizing splines, such as active surfaces, have been used to carry out such tasks, evolving under the influence of internal and external energies until the model converges to a desired surface. The present deformable model based surface extraction techniques are computationally expensive and are generally unreliable in identifying the surfaces of noisy, high-curvature and cluttered 3D objects. This paper proposes a novel decoupled active surface (DAS) for identifying the surface of volumetric 3D objects. The proposed DAS introduces two novel aspects which leads to robust, efficient and accurate convergence. First, rather than a parameterized surface, which leads to difficulties with complex shapes and parameter singularities, the DAS uses a conforming triangular mesh to represent the surface. Second, motivated by earlier successes in two-dimensional segmentation, the DAS treats the two energy components separately and uses novel solution techniques to efficiently minimize the two energy terms separately. The performance of DAS in segmenting static 3D objects is presented using several natural and synthetic volumetric images, with excellent convergence results.

Recovering 3D structure from multiple 2D images is a fundamental problem in computer vision. Most of existing methods focus on the reconstruction of static points in 3D space; however, the reconstruction of trajectories which are resulted from moving points should also have our full attention due to its high efficiency in structure modeling and description. Depending on whether points are moving in spatial domain or in time domain, trajectory recovery turns out to be a curve reconstruction problem or a non-rigid structure recovery problem respectively. This thesis addresses several issues that were not considered in existing approaches in both of the two problems.
\nFor the curve reconstruction problem, we propose a dedicated method for planar curve reconstruction and an optimization method for general curve reconstruction. In the planar curve reconstruction method, measured projected curves that are typically represented by sequences of points are fitted using B-splines before reconstruction, enabling the occlusion problem to be handled naturally. Also, an optimization algorithm is developed to match the fitted curves across images while enforcing the planarity constraint, and the algorithm is guaranteed to converge. In the general curve reconstruction method, Non-Uniform Rational B-Spline (NURBS) is employed for curve representation in 3D space, which improves the flexibility in curve description while maintaining the smoothness of a curve at the same time. Starting with measured point sequences of projected curves, a complete set of algorithms are developed and evaluated, including curve initialization and optimization of the initialized curve by minimizing the 2D reprojection error that is defined to be the 2D Euclidean distance from measured points to reprojected curves. Experiments show that the proposed methods are robust and efficient, and are excellent in producing high-quality reconstruction results. 
\nFor the non-rigid structure recovery problem, we proposed two methods for the recovery of non-rigid structures together with a strategy that automates the process of non-rigid structure recovery. Compared with existing methods using synthetic datasets, both of the two proposed methods perform significantly better than existing methods when there are noise contaminations in measurements, and are capable to recover the ground truth solution when the measurements are noise free whereas no existing method is capable of achieving this so far. In the first method, namely factorization-based method, the available constraints in non-rigid structure from motion are analyzed and the ambiguity of the solution space of the proposed method is clarified, leading to a straightforward approach that requires only solution to several linear equations in least-squares sense instead of having to solve non-linear optimization problems in existing methods. In the second method, namely bundle adjustment method, a modified trajectory basis model that is demonstrated to be more flexible for non-rigid structure description is proposed. The method seeks for optimal non-rigid structure and camera matrices by alternately solving a set of linear equations in least square sense. Experiments on real non-rigid motions show that the method improves the quality of reconstruction significantly.

Image classification is a fundamental problem in computer vision, which deserves much attention in the past decade. To provide a comprehensive recall for the image classification task based on the deep learning algorithms, this paper first provides a brief history of computer vision and convolutional neural networks. Then, the current state of research and the direction of development of deep learning-based convolutional neural network models for image classification are examined. In addition, the basic model structure, convolution feature extraction, and pooling operations of standard and convolutional neural networks are also introduced. This paper summarizes the development of convolutional neural network models in recent years. In summary, based on the review of the development of image classification algorithms at home and abroad, the current mainstream image classification algorithms and frontier progress are summarized and analyzed, and the existing problems and future development directions of image classification are summarized and prospected. It can be concluded that deep learning-based methods has a great potential in this case.

Shape-from-shading (SFS) is an important method to reconstruct three-dimensional (3D) shape of a surface in photometry and computer vision. Lambertian surface reflectance and orthographic camera projection are two fundamental assumptions which generally result in undesirable reconstructed results since inaccurate imaging model is adopted. In this paper, we propose a new fast 3D shape reconstruction approach via the SFS method relaxing the two assumptions. To this end, the Oren-Nayar reflectance and perspective projection models are used to establish an image irradiance equation which depicts the relationship between the 3D shape of non-Lambertian surfaces and its corresponding two-dimensional (2D) shading image. Considering the light attenuation of the near point source, the image irradiance equation is transformed into a static Hamilton-Jacobi partial differential equation (PDE) by solving a quadratic equation. The viscosity solution of the resultant Hamilton-Jacobi PDE is approximated by using optimal control theory and iterative fast marching method starting from a viscosity supersolution. The performance of the proposed approach is evaluated on both synthetic and real-world images and the experimental results demonstrate that the proposed approach is accurate and fast.

Maximum flow and minimum s-t cut algorithms are used to solve several fundamental problems in computer vision. These problems have special structure, and standard techniques perform worse than the special-purpose Boykov-Kolmogorov (BK) algorithm. We introduce the incremental breadth-first search (IBFS) method, which uses ideas from BK but augments on shortest paths. IBFS is theoretically justified (runs in polynomial time) and usually outperforms BK on vision problems.

Novel view synthesis is a classic problem in computer vision. It refers to the generation of previously unseen views of a scene from a set of sparse input images taken from different viewpoints. One example of novel view synthesis is the interpolation of views in between the two images of a stereo camera. Another classic problem in computer vision is video frame interpolation, which is important for video processing. It refers to the generation of video frames in between existing ones and is commonly used to increase the frame rate of a video or to match the frame rate to the refresh rate of the monitor that the video is being displayed on. Interestingly, off-the-shelf video frame interpolation can directly be employed to successfully perform view interpolation to address the aforementioned stereo view interpolation problem. Video frame interpolation can be seen as temporal novel view synthesis. However, this perspective is usually not considered and novel view synthesis generally concerns generating unseen views in space rather than time. For this reason, the set of sparse input images that is used for spatial novel view synthesis is commonly either captured at the same time, or it is assumed that the scene is static. This paradigm limits the applicability of novel view synthesis in real-world scenarios though. This thesis addresses three applications of novel view synthesis and provides practical solutions that do not require difficult-to-acquire multi-view imagery: video frame interpolation which performs temporal video-to-video synthesis, synthesizing the 3D Ken Burns effect from a single image which performs spatial image-to-video synthesis, synthesizing video action shots which performs spatiotemporal video-to-video and video-to-image synthesis. These applications not only explore different dimensions of time and space, they also perform novel view synthesis on everyday image and video footage. This is in stark contrast to the large body of existing work which focuses on spatial novel view synthesis while requiring multiple input views that were either captured at the same time or under the assumption of a static scene.

Shape-adapted smoothing in estimation of 3-D shape cues from affine deformations of local 2-D brightness structure

One of the fundamental problems of computer vision is to extract 3D shape and motion from images. This can be achieved when a scene or object is observed from different viewpoints or over a period of time. There is a wide range of applications, ranging from digitizing cultural heritage to vision-based autonomous robot navigation. This talk will present several approaches to solve this problem. First, we’ll discuss techniques for 3D shape recovery for static objects and scenes. One particular case is the 3D mapping and localization in large environments from images, e.g. urban 3D reconstruction from vehicleborne cameras or localization from cell-phone images. Next, we’ll shift our focus to modeling dynamic scenes, e.g. people who are moving around. In addition to explicitly 3D modeling an event, we'll consider the possibility to perform video-based rendering from casually captured videos. Speaker Bio-Sketch: Marc Pollefeys is a full professor in the Dept. of Computer Science of ETH Zurich since 2007 where he is the head of the Institute for Visual Computing and leads the Computer Vision and Geometry lab. He currently also remains associated with the Dept. of Computer Science of the University of North Carolina at Chapel Hill where he started as an assistant professor in 2002 and became an associate professor in 2005. Before this he was a postdoctoral researcher at the Katholieke Universiteit Leuven in Belgium, where he also received his M.S. and Ph.D. degrees in 1994 and 1999, respectively. His main area of research is computer vision. One of his main research goals is to develop flexible approaches to capture visual representations of real world objects, scenes and events. Dr. Pollefeys has received several prizes for his research, including a Marr prize, an NSF CAREER award, a Packard Fellowship and a European Research Council Starting Grant. He is the author or co-author of more than 130 peer-reviewed publications. He is the General Chair for the European Conference on Computer Vision 2014 (ECCV), was a Program Co-Chair for the IEEE Conference on Computer Vision and Pattern Recognition 2009 (CVPR), was general/program co-chair of the Third Symposium on 3D Data Processing, Visualization and Transmission and has organized workshops and courses at major vision and graphics conferences and has served on the program committees of many conferences. Prof. Pollefeys is/was on the Editorial Board of the IEEE Transactions on Pattern Analysis and Machine Intelligence and the International Journal of Computer Vision as well as several other journals in computer vision, graphics and robotics.

Homography estimation is a fundamental problem in computer vision. Previous works mainly focus on estimating either a single homography, or multiple homographies based on mesh grid division of the image. In practical scenarios, single homography is inadequate and often leads to a compromised result for multiple planes; while mesh grid multi-homography damages the plane distribution of the scene, and does not fully address the restriction to use homography. In this work, we propose a novel semantics guided multi-homography estimation framework, Mask-Homo, to provide an explicit solution to the multi-plane depth disparity problem. First, a pseudo plane mask generation module is designed to obtain multiple correlated regions that follow the plane distribution of the scene. Then, multiple local homography transformations, each of which aligns a correlated region precisely, are predicted and corresponding warped images are fused to obtain the final result. Furthermore, a new metric, Mask-PSNR, is proposed for more comprehensive evaluation of alignment. Extensive experiments are conducted to verify the effectiveness of the proposed method. Our code is available at https://github.com/SAITPublic/MaskHomo.