Optimizing polynomial solvers for minimal geometry problems

Novel view synthesis is a classic problem in computer vision. It refers to the generation of previously unseen views of a scene from a set of sparse input images taken from different viewpoints. One example of novel view synthesis is the interpolation of views in between the two images of a stereo camera. Another classic problem in computer vision is video frame interpolation, which is important for video processing. It refers to the generation of video frames in between existing ones and is commonly used to increase the frame rate of a video or to match the frame rate to the refresh rate of the monitor that the video is being displayed on. Interestingly, off-the-shelf video frame interpolation can directly be employed to successfully perform view interpolation to address the aforementioned stereo view interpolation problem. Video frame interpolation can be seen as temporal novel view synthesis. However, this perspective is usually not considered and novel view synthesis generally concerns generating unseen views in space rather than time. For this reason, the set of sparse input images that is used for spatial novel view synthesis is commonly either captured at the same time, or it is assumed that the scene is static. This paradigm limits the applicability of novel view synthesis in real-world scenarios though. This thesis addresses three applications of novel view synthesis and provides practical solutions that do not require difficult-to-acquire multi-view imagery: video frame interpolation which performs temporal video-to-video synthesis, synthesizing the 3D Ken Burns effect from a single image which performs spatial image-to-video synthesis, synthesizing video action shots which performs spatiotemporal video-to-video and video-to-image synthesis. These applications not only explore different dimensions of time and space, they also perform novel view synthesis on everyday image and video footage. This is in stark contrast to the large body of existing work which focuses on spatial novel view synthesis while requiring multiple input views that were either captured at the same time or under the assumption of a static scene.

Gender classification is one of the most challenging problems in computer vision. Facial gender detection of neonates and children is also known as a highly demanding issue for human observers. This study proposes a novel gender classification method using frontal facial images of people. The proposed approach employs principal component analysis (PCA) and fuzzy clustering technique, respectively, for feature extraction and classification steps. In other words, PCA is applied to extract the most appropriate features from images as well as reducing the dimensionality of data. The extracted features are then used to assign the new images to appropriate classes – male or female – based on fuzzy clustering. The computational time and accuracy of the proposed method are examined together and the prominence of the proposed approach compared to most of the other well‐known competing methods is proved, especially for younger faces. Experimental results indicate the considerable classification accuracies which have been acquired for FG‐Net, Stanford and FERET databases. Meanwhile, since the proposed algorithm is relatively straightforward, its computational time is reasonable and often less than the other state‐of‐the‐art gender classification methods.

The simultaneous recovery of three-dimensional (3D) structure from motion (SfM) for the sequences of images, is one of the more difficult problems in computer vision. Classical approaches to the problem rely on using algebraic techniques to solve for these unknowns given two or more image. Motion analysis and 3D shape estimation based on the estimated motion is an important problem in computer vision. The correspondence problem is an important tool in SfM where in this paper a general 3D motion based on a simple rotation, tilt, roll and translation is proposed, and then is used for 3D shape estimation. The current work expands the 2D motion estimation (shift, rotation) to accommodate general 3D motion (shift, rotation, tilt and roll). The proposed work in this paper of motion estimation of moving object is based on Fourier Descriptor Transformation (FDT) analysis before and after motion. The FDT is also used to resolve the correspondence problem in a sequence of images. We test our method on several large-scale photo collections, show the efficacy of the introduced approach to improve reconstruction accuracy.

Graph and hypergraph matching are fundamental problems in computer vision, which have been receiving a steadily growing attention in the last two decades due to the recent modern algorithms and models that are both fast and accurate. Graph and hypergraph matching are successful in tackling many real-world tasks that require 2D and 3D feature matching and alignment. Matching is a general problem in vision and graphs have always been quintessential for modeling information that has both local and longer range connections. Therefore, graph matching will continue to influence computer vision and robotics research, crossing over many different mathematical and computational models, including Bayesian networks, conditional and Markov random fields, spectral graph models, and, more recently, deep neural networks. While graph matching is limited to using pairwise relationships, hypergraph matching permits the use of relationships between sets of features of any order. Consequently, it carries the promise to make matching more robust to changes in scale, deformations, and outliers. In this chapter, we present several key ideas, computational models, and optimization methods that introduced in the literature the possibility to learn graph and hypergraph matching in a semi-supervised or even completely unsupervised fashion. We start by describing two methods for solving graph and hypergraph matching based on spectral matching and integer projected fixed point methods, introduced in Chap. 1. Then, based on the same intuition (also discussed in Chap. 1) that correct assignments form a strong cluster of agreements, we follow up by introducing the main approach to learning as it applies to graph and hypergraph matching, as well as the more general Maximum A Posteriori (MAP) inference problem in probabilistic graphical models (MRFs and CRFs). In the last part of the chapter, we demonstrate experimentally the effectiveness of the algorithms presented and, whenever possible, provide extensive comparisons to other top published methods. Moreover, we show that our learning approach is general and can significantly boost the performance of different state-of-the-art matching algorithms in the literature.

Toward designing intelligent PDEs for computer vision: An optimal control approach

Acquiring qualified samples for RANSAC using geometrical constraints

In the areas of computer vision and computer graphics, analyzing information about three-dimensional objects from two-dimensional images is essential. Shape from shading is a very fundamental problem in computer vision, which uses features and patterns in images to infer the shape of the object in our viewing direction. The influence of solving this problem is significant. For example, a typical application of shape from shading is in astronomy, in which we can obtain landscape features of a planet by analyzing the information from the image we get.

Which parts of an image evoke emotions in an observer? To answer this question, we introduce a novel problem in computer vision — predicting an Emotion Stimuli Map (ESM), which describes pixel-wise contribution to evoked emotions. Building a new image database, EmotionROI, as a benchmark for predicting the ESM, we find that the regions selected by saliency and objectness detection do not correctly predict the image regions which evoke emotion. Although objects represent important regions for evoking emotion, parts of the background are also important. Based on this fact, we propose using fully convolutional networks for predicting the ESM. Both qualitative and quantitative experimental results confirm that our method can predict the regions which evoke emotion better than both saliency and objectness detection.

One of the fundamental problems of computer vision is to extract 3D shape and motion from images. This can be achieved when a scene or object is observed from different viewpoints or over a period of time. There is a wide range of applications, ranging from digitizing cultural heritage to vision-based autonomous robot navigation. This talk will present several approaches to solve this problem. First, we’ll discuss techniques for 3D shape recovery for static objects and scenes. One particular case is the 3D mapping and localization in large environments from images, e.g. urban 3D reconstruction from vehicleborne cameras or localization from cell-phone images. Next, we’ll shift our focus to modeling dynamic scenes, e.g. people who are moving around. In addition to explicitly 3D modeling an event, we'll consider the possibility to perform video-based rendering from casually captured videos. Speaker Bio-Sketch: Marc Pollefeys is a full professor in the Dept. of Computer Science of ETH Zurich since 2007 where he is the head of the Institute for Visual Computing and leads the Computer Vision and Geometry lab. He currently also remains associated with the Dept. of Computer Science of the University of North Carolina at Chapel Hill where he started as an assistant professor in 2002 and became an associate professor in 2005. Before this he was a postdoctoral researcher at the Katholieke Universiteit Leuven in Belgium, where he also received his M.S. and Ph.D. degrees in 1994 and 1999, respectively. His main area of research is computer vision. One of his main research goals is to develop flexible approaches to capture visual representations of real world objects, scenes and events. Dr. Pollefeys has received several prizes for his research, including a Marr prize, an NSF CAREER award, a Packard Fellowship and a European Research Council Starting Grant. He is the author or co-author of more than 130 peer-reviewed publications. He is the General Chair for the European Conference on Computer Vision 2014 (ECCV), was a Program Co-Chair for the IEEE Conference on Computer Vision and Pattern Recognition 2009 (CVPR), was general/program co-chair of the Third Symposium on 3D Data Processing, Visualization and Transmission and has organized workshops and courses at major vision and graphics conferences and has served on the program committees of many conferences. Prof. Pollefeys is/was on the Editorial Board of the IEEE Transactions on Pattern Analysis and Machine Intelligence and the International Journal of Computer Vision as well as several other journals in computer vision, graphics and robotics.

No object is mysterious. The mystery is your eye. – Elizabeth Bowen Introduction There are two fundamentally different philosophies concerning understanding the brain. (1) Understand the brain first. If we can understand how the brain works, we can build smart machines. (2) Using any technique we can think of, make a smart machine. If we can accomplish that, it will give us some hints about how the brain works. This book is all about the second approach, although it draws from current understanding of biological computing. In this chapter, however, we define a few terms, introduce the greater localglobal problem, and then give a very brief introduction to the function of the mammalian brain. • (Section 1.2) From signal and systems perspective, we describe the differences between Computer Vision and some other closely related fields of studies, including, e.g., image processing and pattern recognition. • (Section 1.3) Since almost all problems in Computer Vision involve the issue of localness versus globalness, we briefly explain the “local-global” problem and the “consistency” principle used to solve this problem. • (Section 1.4) Computer Vision is deep-rooted in biological vision. Therefore, in this section, we discuss the biological motivation of Computer Vision and some amazing discoveries from the study of the human visual system. Some Definitions Computer Vision is the process whereby a machine, usually a digital computer, automatically processes an image and reports “what is in the image.” That is, it recognizes the content of the image. For example, the content may be a machined part, and the objective may be not only to locate the part but to inspect it as well. Students tend to get confused by other terms that often appear in the literature, such as Image Processing , Machine Vision , Image Understanding , and Pattern Recognition . We can divide the entire process of Image Processing into Low-Level Image Processing and High-Level Image Processing . If we interpret these processes from signal and systems perspective, it is more clear to describe their difference and similarity from the format of input/output of the system. When a Low-Level Image Processing system processes an input image, the output is still an image, but a somewhat different image. For example, it may be an image with noise removed, an image that does not take as much storage space as the input image, an image that is sharper than the input image, etc.

A wide range of problems in computer graphics and vision can be formulated as sparse least squares problems. For example, Laplacian mesh deformation, Least Squares Conformal Maps, Poisson image editing, and as-rigid-as-possible (ARAP) image warping involve solving a linear or non-linear sparse least squares problem. High performance is crucial in many of these applications for interactive user feedback. For these applications, we show that the matrices produced by factorization methods such as QR have a special structure: the off-diagonal blocks are low-rank. We leverage this property to produce a fast sparse approximate QR factorization, spaQR, for these matrices in near-linear time. In our benchmarks, spaQR shows up to 57% improvement over solving the normal equations using Cholesky and 63% improvement over a standard preconditioner with Conjugate Gradient Least Squares (CGLS).

QR and LU decompositions are the most important matrix decomposition algorithms. Many studies work on accelerating these algorithms by FPGA or ASIC in a case by case style. In this paper, we propose a unified framework for the matrix decomposition algorithms, combining three QR decomposition algorithms and LU algorithm with pivoting into a unified linear array structure. The QR and LU decomposition algorithms exhibit the same two-level loop structure and the same data dependency. Utilizing the similarities in loop structure and data dependency of matrix decomposition, we unify a fine-grained algorithm for all four matrix decomposition algorithms. Furthermore, we present a unified coprocessor structure with a scalable linear array of processing elements (PEs), in which four types of PEs are same in the structure of memory channels and PE connections, but the only difference exists in the internal structure of data path. Our unified co-processor, which is IEEE 32-bit floating-point precision, is implemented and mapped onto a Xilinx Virtex5 FPGA chip. Experimental results show that our co-processors can achieve speedup of 2.3 to 14.9 factors compared to a Pentium Dual CPU with double SSE threads.

Estimating the 6-DoF camera pose of an image with respect to a 3D scene model, known as visual localization, is a fundamental problem in many computer vision and robotics tasks. Among various visual localization methods, the direct 2D-3D matching method has become the preferred method for many practical applications due to its computational efficiency. When using direct 2D-3D matching methods in large-scale scenes, a vocabulary tree can be used to accelerate the matching process, which will also induce the quantization artifacts leading to reduce the inlier ratio and decrease the localization accuracy. To this end, in this paper two simple and effective mechanisms, called visibility-based recalling and space-based recalling, are proposed to recover lost matches caused by the quantization artifacts, thus can largely improve the localization accuracy and success rate without increasing too much computational time. Experimental results on long-term visual localization benchmarks demonstrate the effectiveness of our method compared with state-of-the-arts.

The article considers the possibilities of using the deep learning convolutional neural network ResNet in computer vision and image classification problems. The interpretation of the ResNet network and the datasets used for its training are presented, as well as a method for training a deep convolutional neural network with stochastic depth, which allows significantly reducing errors in the test sample.

How to Track a Flying Saucer