Camera internal parameter calibration based on rotating platform and image matching

An improved outdoor camera calibration method based on Extended Kalman Filter (EKF) is proposed in this paper. GPS coordinate system is introduced to establish world coordinate system in the calibration procedure. According to the characteristics of GPS coordinate converted to plane coordinate and the camera calibration model, we establish the mapping relations of reference points between image plane coordinate and the world coordinate. Further, we consider the world coordinate error, set the world coordinates as the measure quantities, use the method of EKF to get the optimal estimation of intrinsic and extrinsic camera parameters, in EKF the camera's intrinsic and extrinsic parameters are set as state variables, the transformation equations of image coordinate to the world coordinate are set as the measurement equations, and we let the observed feature points' image coordinates and corresponding world coordinates as the input of the filter. Experimental results confirmed that the proposed method with high precision. Index Terms - camera calibration, GPS coordinate, EKF, world coordinate error

This chapter presents a method for image matching which is based on the detection and characterisation of curvilinear regions. In a curvilinear region, the ratio between its average width and its total length should be less than a predefined threshold. Besides, left and right borders should be locally parallel, it should exist a geometric similarity around the region axis and the colour along this axis should be homogeneous. That is, they constitute particular image structures and, therefore, the method is restricted to scenes where these particular items are presented. On the contrary, curvilinear regions automatically deform with changing viewpoint as to keep on covering identical physical parts of a scene. For this reason, they can be used as distinguished regions. The shape contour of these curvilinear regions is characterised using the adaptive curvature function. Experimental results show that this shape descriptor is invariant to rotation and translation, and partially invariant to noise and skewing. Scaling invariance is achieved by employing an extra normalisation stage. Thus, this descriptor, plus the region colour and position, can be used to match curvilinear regions detected on the input image with those previously stored in a database. This is the basis of the correspondence algorithm described in this paper: the similarity index between two images is determined by the presence of the same set of curvilinear regions localised in similar positions. There are many directions for further research. One of this is the integration of several types of distinguished regions. As we commented above, the obligatory presence of these regions in the images is the main disadvantage of the proposed system. Besides, further work must be accomplished in the correspondence algorithm to employ the most similar region correspondences as ground control points. These points could be used to generate a fundamental matrix describing the geometric constraints between the two images. Thus, matches that did not agree with the majority solution could be removed.

Image matching is a fundamental task of many problems in computer vision. This paper presents a novel local feature descriptor based on the gradient distance and orientation histogram (GDOH), which can be used for reliably matching between different views of a scene for wide baseline. The proposed descriptor is invariant to image scale, rotation, illumination and partial viewpoint changes. At present, the SIFT descriptor is generally considered as the most appealing descriptor for practical uses, but the high dimensionality is a drawback of SIFT in the feature matching step. The purpose of GDOH is to reduce the dimensional size of the descriptor, yet still maintain distinctness and robustness as much as SIFT. The experimental results show that the proposed descriptor can result in effectiveness and efficiency in image matching and image retrieval application.

The algorithms based on image feature detection and matching are critical in the field of computer vision. Feature extraction and matching algorithms are used in many computer vision problems, including object recognition and structure from motion. Each feature detector and descriptor algorithm’s computational efficiency and robust performance have a major impact on image matching precision and time utilization. The performance of image matching algorithms that use expensive descriptors for detection and matching is addressed using existing approaches. The algorithm’s efficiency is measured by the number of matches found and the number of faults discovered when evaluated against a given pair of images. It also depends on the algorithm that detects the features and matches them in less amount of time. This paper examines and compares the different algorithm (SURF, ORB, BRISK, FAST, KAZE, MINEIGEN, MSER) performances using distinct parameters such as affine transformation, blur, scale, illumination, and rotation. The Oxford dataset is used to assess their robustness and efficiency against the parameters of interest. The time taken to detect features, the time taken to match images, the number of identified feature points, and the total running time is recorded in this study. The quantitative results show that the ORB and SURF algorithms detect and match more features than other algorithms. Furthermore, they are computationally less expensive and robust compared to other algorithms. In addition, the robustness of ORB and SURF is quite high in terms of outliers, and the amount of time taken to match with the reference is also significantly less. However, the efficiency of SURF reduces against blur transformation. FAST is good in detecting corners but lacks efficiency under different transformations. Experiments show that when each algorithm is subjected to numerous alterations, it has its own set of advantages and limitations.

Estimating the pose of a plane given a set of point correspondences is a core problem in computer vision with many applications including Augmented Reality (AR), camera calibration and 3D scene reconstruction and interpretation. Despite much progress over recent years there is still the need for a more efficient and more accurate solution, particularly in mobile applications where the run-time budget is critical. We present a new analytic solution to the problem which is far faster than current methods based on solving Pose from $$n$$ n Points (PnP) and is in most cases more accurate. Our approach involves a new way to exploit redundancy in the homography coefficients. This uses the fact that when the homography is noisy it will estimate the true transform between the model plane and the image better at some regions on the plane than at others. Our method is based on locating a point where the transform is best estimated, and using only the local transformation at that point to constrain pose. This involves solving pose with a local non-redundant 1st-order PDE. We call this framework Infinitesimal Plane-based Pose Estimation (IPPE), because one can think of it as solving pose using the transform about an infinitesimally small region on the surface. We show experimentally that IPPE leads to very accurate pose estimates. Because IPPE is analytic it is both extremely fast and allows us to fully characterise the method in terms of degeneracies, number of returned solutions, and the geometric relationship of these solutions. This characterisation is not possible with state-of-the-art PnP methods.

A unified feature-spatial cycle consistency fusion framework for robust image matching

High-dimensional image feature matching is an important part of many image matching based problems in computer vision which are solved by local invariant features. In this paper, we propose a new indexing/searching method based on Randomized Sub-Vectors Hashing (called RSVH) for high-dimensional image feature matching. The essential of the proposed idea is that the feature vectors are considered similar (measured by Euclidean distance) when the L2 norms of their corresponding randomized sub-vectors are approximately same respectively. Experimental results have demonstrated that our algorithm can perform much better than the famous BBF (Best-Bin-First) and LSH (Locality Sensitive Hashing) algorithms in extensive image matching and image retrieval applications.

Novel view synthesis is a classic problem in computer vision. It refers to the generation of previously unseen views of a scene from a set of sparse input images taken from different viewpoints. One example of novel view synthesis is the interpolation of views in between the two images of a stereo camera. Another classic problem in computer vision is video frame interpolation, which is important for video processing. It refers to the generation of video frames in between existing ones and is commonly used to increase the frame rate of a video or to match the frame rate to the refresh rate of the monitor that the video is being displayed on. Interestingly, off-the-shelf video frame interpolation can directly be employed to successfully perform view interpolation to address the aforementioned stereo view interpolation problem. Video frame interpolation can be seen as temporal novel view synthesis. However, this perspective is usually not considered and novel view synthesis generally concerns generating unseen views in space rather than time. For this reason, the set of sparse input images that is used for spatial novel view synthesis is commonly either captured at the same time, or it is assumed that the scene is static. This paradigm limits the applicability of novel view synthesis in real-world scenarios though. This thesis addresses three applications of novel view synthesis and provides practical solutions that do not require difficult-to-acquire multi-view imagery: video frame interpolation which performs temporal video-to-video synthesis, synthesizing the 3D Ken Burns effect from a single image which performs spatial image-to-video synthesis, synthesizing video action shots which performs spatiotemporal video-to-video and video-to-image synthesis. These applications not only explore different dimensions of time and space, they also perform novel view synthesis on everyday image and video footage. This is in stark contrast to the large body of existing work which focuses on spatial novel view synthesis while requiring multiple input views that were either captured at the same time or under the assumption of a static scene.

Graph and hypergraph matching are fundamental problems in computer vision, which have been receiving a steadily growing attention in the last two decades due to the recent modern algorithms and models that are both fast and accurate. Graph and hypergraph matching are successful in tackling many real-world tasks that require 2D and 3D feature matching and alignment. Matching is a general problem in vision and graphs have always been quintessential for modeling information that has both local and longer range connections. Therefore, graph matching will continue to influence computer vision and robotics research, crossing over many different mathematical and computational models, including Bayesian networks, conditional and Markov random fields, spectral graph models, and, more recently, deep neural networks. While graph matching is limited to using pairwise relationships, hypergraph matching permits the use of relationships between sets of features of any order. Consequently, it carries the promise to make matching more robust to changes in scale, deformations, and outliers. In this chapter, we present several key ideas, computational models, and optimization methods that introduced in the literature the possibility to learn graph and hypergraph matching in a semi-supervised or even completely unsupervised fashion. We start by describing two methods for solving graph and hypergraph matching based on spectral matching and integer projected fixed point methods, introduced in Chap. 1. Then, based on the same intuition (also discussed in Chap. 1) that correct assignments form a strong cluster of agreements, we follow up by introducing the main approach to learning as it applies to graph and hypergraph matching, as well as the more general Maximum A Posteriori (MAP) inference problem in probabilistic graphical models (MRFs and CRFs). In the last part of the chapter, we demonstrate experimentally the effectiveness of the algorithms presented and, whenever possible, provide extensive comparisons to other top published methods. Moreover, we show that our learning approach is general and can significantly boost the performance of different state-of-the-art matching algorithms in the literature.

Recovering 3D structure from multiple 2D images is a fundamental problem in computer vision. Most of existing methods focus on the reconstruction of static points in 3D space; however, the reconstruction of trajectories which are resulted from moving points should also have our full attention due to its high efficiency in structure modeling and description. Depending on whether points are moving in spatial domain or in time domain, trajectory recovery turns out to be a curve reconstruction problem or a non-rigid structure recovery problem respectively. This thesis addresses several issues that were not considered in existing approaches in both of the two problems.
\nFor the curve reconstruction problem, we propose a dedicated method for planar curve reconstruction and an optimization method for general curve reconstruction. In the planar curve reconstruction method, measured projected curves that are typically represented by sequences of points are fitted using B-splines before reconstruction, enabling the occlusion problem to be handled naturally. Also, an optimization algorithm is developed to match the fitted curves across images while enforcing the planarity constraint, and the algorithm is guaranteed to converge. In the general curve reconstruction method, Non-Uniform Rational B-Spline (NURBS) is employed for curve representation in 3D space, which improves the flexibility in curve description while maintaining the smoothness of a curve at the same time. Starting with measured point sequences of projected curves, a complete set of algorithms are developed and evaluated, including curve initialization and optimization of the initialized curve by minimizing the 2D reprojection error that is defined to be the 2D Euclidean distance from measured points to reprojected curves. Experiments show that the proposed methods are robust and efficient, and are excellent in producing high-quality reconstruction results. 
\nFor the non-rigid structure recovery problem, we proposed two methods for the recovery of non-rigid structures together with a strategy that automates the process of non-rigid structure recovery. Compared with existing methods using synthetic datasets, both of the two proposed methods perform significantly better than existing methods when there are noise contaminations in measurements, and are capable to recover the ground truth solution when the measurements are noise free whereas no existing method is capable of achieving this so far. In the first method, namely factorization-based method, the available constraints in non-rigid structure from motion are analyzed and the ambiguity of the solution space of the proposed method is clarified, leading to a straightforward approach that requires only solution to several linear equations in least-squares sense instead of having to solve non-linear optimization problems in existing methods. In the second method, namely bundle adjustment method, a modified trajectory basis model that is demonstrated to be more flexible for non-rigid structure description is proposed. The method seeks for optimal non-rigid structure and camera matrices by alternately solving a set of linear equations in least square sense. Experiments on real non-rigid motions show that the method improves the quality of reconstruction significantly.

A primal sketch based framework for bean-shape contour extraction

Toward designing intelligent PDEs for computer vision: An optimal control approach

Hand tracking is widely used in virtual reality systems and it is still a critical problem in computer vision and graphics. In this paper, a structured light scanning based hand tracking system is proposed. The pattern designed in the real-time scanning system is composed of color stripes with quadratic intensity distribution in each stripe. After camera and projector calibration, 3D data of a moving hand can be acquired using the real-time system. For each frame of the sequence, the fingertips are detected by making use of the boundary of the 3D precise data. In comparison with the methods based on passive vision only, the proposed system is more robust and suitable for bare hand tracking. Some experiments have been performed to evaluate the effectiveness of the proposed method.

This paper presents an algorithm for automated extraction of interest points (IPs)in multispectral and hyperspectral images. Interest points are features of the image that capture information from its neighbours and they are distinctive and stable under transformations such as translation and rotation. Interest-point operators for monochromatic images were proposed more than a decade ago and have since been studied extensively. IPs have been applied to diverse problems in computer vision, including image matching, recognition, registration, 3D reconstruction, change detection, and content-based image retrieval. Interest points are helpful in data reduction, and reduce the computational burden of various algorithms (like registration, object detection, 3D reconstruction etc) by replacing an exhaustive search over the entire image domain by a probe into a concise set of highly informative points. An interest operator seeks out points in an image that are structurally distinct, invariant to imaging conditions, stable under geometric transformation, and interpretable which are good candidates for interest points. Our approach extends ideas from Lowe's keypoint operator that uses local extrema of Difference of Gaussian (DoG) operator at multiple scales to detect interest point in gray level images. The proposed approach extends Lowe's method by direct conversion of scalar operations such as scale-space generation, and extreme point detection into operations that take the vector nature of the image into consideration. Experimental results with RGB and hyperspectral images which demonstrate the potential of the method for this application and the potential improvements of a fully vectorial approach over band-by-band approaches described in the literature.

In the areas of computer vision and computer graphics, analyzing information about three-dimensional objects from two-dimensional images is essential. Shape from shading is a very fundamental problem in computer vision, which uses features and patterns in images to infer the shape of the object in our viewing direction. The influence of solving this problem is significant. For example, a typical application of shape from shading is in astronomy, in which we can obtain landscape features of a planet by analyzing the information from the image we get.