Performance evaluation of stereo for tele-presence

This paper presents live and interactive webware for online learning of computer graphics concepts. A list of demos is provided. Each demo presents a concept in computer graphics by showing a 3D real world scene beside a 2D rendering scene with a list of graphics functions. Each demo allows users to interactively change the parameters and the order of execution of these graphics functions. Changing the parameters of the functions will produce the 2D rendering result from the 3D real world scene. The visual effects of user interaction will be reflected immediately in the 3D real world scene and the 2D rendering result. The webware was written by using the GL4Java library that provides native OpenGL binding for Java. Nate Robin’s well-known demos were implemented. These include translation, projection, light effect, texture mapping, and so on. New demos were also developed with pedagogical considerations in mind to emphasize the differences between model transformation and view transformation. Although the webware is designed for computer graphics learning the methodology is generic and can easily be applied to other disciplines or courses that require heavy visual presentation. This webware reflects our long-term efforts to develop web-based course material to show principles and techniques in computer science in an interactive way. We did this by having the related algorithms run live in the background and allowing students to interact with them in a web browser.

As a frontier technology, holography has important research values in fields such as bio-micrographic imaging, light field modulation and data storage. However, the real-time acquisition of 3D scenes and high-fidelity reconstruction technology has not yet made a breakthrough, which has seriously hindered the development of holography. Here, a novel holographic camera is proposed to solve the above inherent problems completely. The proposed holographic camera consists of the acquisition end and the calculation end. At the acquisition end of the holographic camera, specially configured liquid materials and liquid lens structure based on voice-coil motor-driving are used to produce the liquid camera, so that the liquid camera can quickly capture the focus stack of the real 3D scene within 15 ms. At the calculation end, a new structured focus stack network (FS-Net) is designed for hologram calculation. After training the FS-Net with the focus stack renderer and learnable Zernike phase, it enables hologram calculation within 13 ms. As the first device to achieve real-time incoherent acquisition and high-fidelity holographic reconstruction of a real 3D scene, our proposed holographic camera breaks technical bottlenecks of difficulty in acquiring the real 3D scene, low quality of the holographic reconstructed image, and incorrect defocus blur. The experimental results demonstrate the effectiveness of our holographic camera in the acquisition of focal plane information and hologram calculation of the real 3D scene. The proposed holographic camera opens up a new way for the application of holography in fields such as 3D display, light field modulation, and 3D measurement.

We analyze Kinect as a 3D measuring device, experimentally investigate depth measurement resolution and error properties and make a quantitative comparison of Kinect accuracy with stereo reconstruction from SLR cameras and a 3D-TOF camera. We propose Kinect geometrical model and its calibration procedure providing an accurate calibration of Kinect 3D measurement and Kinect cameras. We demonstrate the functionality of Kinect calibration by integrating it into an SfM pipeline where 3D measurements from a moving Kinect are transformed into a common coordinate system by computing relative poses from matches in color camera.

In images with low texture the performance of conventional dense stereo can be poor. The usual solution to this is to use a large window but this itself can be problematic as the large window can blur important features and hence lead to errors in the disparity estimate. Here it is shown that, not only do connected set morphology operators overcome this problem, they perform best in regions of low texture. A further observation is that, since the operators give a heirarchical decompostion, there is a possibility of not only using these operators to choose a new window, but also to motivate a new matching method.

Underwater 3D object reconstruction is one of the most essential and fundamental tasks in ocean investigations. In this paper, we try to capture the inherent geometrical variation of 3D objects at multiple visual angels with the underwater vehicles and develop a novel underwater 3D object reconstruction model for the continuous video stream. Image enhancement will be first taken into consideration by guided image filtering to acquire clearer images for object tracking which can be completed by the particle filter. And then, we use scale invariant feature transform (SIFT) and random sample consensus (RANSAC) to detect and correspond the features of objects we have tracked. From these correspondences, the 3D point cloud reconstruction will be finally developed with the help of structure from motion (SFM) and patches-based multi-view stereo (PMVS). It has been shown in the simulation experiments that the developed scheme of this paper achieves consistent performance improvements over 3D object reconstruction for underwater video stream in multiple views.

This chapter describes the basic concepts of partial differential equations (PDEs) based image modelling and their applications to image restoration. The general basic concepts of partial differential equation (PDE)-based image modelling and processing techniques are discussed for image restoration problems. These techniques can also be used in the design and development of efficient tools for various image processing and vision related tasks such as restoration, enhancement, segmentation, registration, inpainting, shape from shading, 3D reconstruction of objects from multiple views, and many more. As a case study, the topic in consideration is oriented towards image restoration using PDEs formalism since image restoration is considered to be an important pre-processing task for 3D surface geometry, reconstruction, and many other applications. An image may be subjected to various types of noises during its acquisition leading to degraded quality of the image, and hence, the noise must be reduced. The noise may be additive or multiplicative in nature. Here, the PDE-based models for removal of both types of noises are discussed. As examples, some PDE-based schemes have been implemented and their comparative study with other existing techniques has also been presented.

The detection and recognition of targets obscured by natural or simulated arboreal cover can be facilitated by combining multiple images of the target taken from different viewpoints or look angles. This technique, called multi-look imaging (MLI), has three principal advantages: (1) obscured target features can be revealed in one or more of the views; (2) fusion of information about a given target feature contained in multiple views can yield apparent noise reduction and increased feature resolution; (3) background features can be reduced via correlation and subtraction, thereby decreasing the target-to-clutter ratio. In Part 1 of this series of two paper, we discussed background and theory that support airborne MLI of ground targets. In this paper, we continue theory development with an analysis of geometric projection error inherent in target reconstruction. The effect of projection error derived from focal-plane quantization error during back-projection of a given look or view in model-based target reconstruction is of particular interest, due to our current research emphasis upon ray-projection based reconstruction algorithms. Additional error sources discussed in this paper include uneven coverage of looks, which can yield space-variant uncertainty associated with reconstruction of target features having low information content in relation to other target or background features. Simulation results are based on a calibrated model of airborne MLI, and are analyzed in terms of computational cost and accuracy of target reconstruction achievable via stereophotogrammetric methods.

We present a unified framework for 3-D shape reconstruction that allows us to combine multiple information sources—such as stereo and shape-from-shading—derived from images whose vantage points may be very different. A formal integration framework is critical because, in recovering complicated surfaces, the information from a single source is often insufficient to provide a unique answer.We describe two complementary implementations of this paradigm that both rely on deforming a generic object-centered 3-D representation of the surface so as to minimize an objective function. The first implementation models the surface as a triangulated mesh. The second models it as a set of particles that interact with each other through forces that tend to align them.

We propose a framework for 2D/3D multi-modal data registration and evaluate 3D feature descriptors for registration of 3D datasets from different sources. 3D datasets of outdoor environments can be acquired using a variety of active and passive sensor technologies. Registration of these datasets into a common coordinate frame is required for subsequent modelling and visualisation. 2D images are converted into 3D structure by stereo or multiview reconstruction techniques and registered to a unified 3D domain with other datasets in a 3D world. Multi-modal datasets have different density, noise, and types of errors in geometry. This paper provides a performance benchmark for existing 3D feature descriptors across multi-modal datasets. This analysis highlights the limitations of existing 3D feature detectors and descriptors which need to be addressed for robust multi-modal data registration. We analyse and discuss the performance of existing methods in registering various types of datasets then identify future directions required to achieve robust multi-modal data registration.

In this paper, we describe a new freehand ultrasound imaging system for reconstructing the left ventricle from 2D echocardiography slices. An important contribution of the proposed system is its ability to reconstruct from multiple standard views. The multi-view reconstruction procedure results in significant reduction in reconstruction error over single view reconstructions. The system uses object-based 3D volumetric registration, allowing for arbitrary rigid object movements in inter-view acquisition. Furthermore, a new segmentation procedure that combines level set methods with gradient vector flow(GVF) is used for automatically segmenting the 2D ultrasound images, in which low level of contrast, high level of speckle noise, and weak boundaries are common. The new segmentation approach is shown to be robust to these artifacts and is found to converge to the boundary from a wider range of initial conditions than competitive methods. The proposed system has been validated on a physical, 3D ultrasound calibration phantom and evaluated on one actual cardiac echocardiography data set. In the phantom experiment, two calibrated volumetric egg-shape objects were scanned from the top and side windows and reconstructed using the new method. The volume error was measured to be less than 4%. In a real heart data set experiment, qualitative results of 3D surface reconstruction from parasternal and apical views appear significantly improved over single view reconstructions. The estimated volumes from the 3D reconstructions were also found to be in agreement with the manual clinical measurements from 2D slices. Further extension of this work is to compare the quantitative results with more accuracy MRI data.

In this paper we address the problem of estimating the intrinsic parameters of a 3D LIDAR while at the same time computing its extrinsic calibration with respect to a rigidly connected camera. Existing approaches to solve this nonlinear estimation problem are based on iterative minimization of nonlinear cost functions. In such cases, the accuracy of the resulting solution hinges on the availability of a precise initial estimate, which is often not available. In order to address this issue, we divide the problem into two least-squares sub-problems, and analytically solve each one to determine a precise initial estimate for the unknown parameters. We further increase the accuracy of these initial estimates by iteratively minimizing a batch nonlinear least-squares cost function. In addition, we provide the minimal identifiability conditions, under which it is possible to accurately estimate the unknown parameters. Experimental results consisting of photorealistic 3D reconstruction of indoor and outdoor scenes, as well as standard metrics of the calibration errors, are used to assess the validity of our approach.

Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding

In this paper we describe a technique of fusing larger scenes from multiple views, using a landmark approach. For images of remote scenes, the fused image is reconstructed using sample mean shapes on affine shape manifolds. This manifold of configurations of points in general position is a real Grassmann manifold which can be equivariantly embedded in a space of matrices, and we consider the extrinsic means w.r.t. this embedding. On the other hand, for photographs taken from a close distance by a noncalibrated camera, the fused image is reconstructed using sample mean shapes on projective shape manifolds. This manifold of configurations of points under a given projective frame is a product of real projective planes and we consider extrinsic means w.r.t. to a standard embedding in a space of blocks of symmetric matrices. We illustrate these methods on two sets of images.

We propose a method for combining geometric and real-aperture methods for monocular 3D reconstruction of static scenes at absolute scales. Our algorithm relies on a sequence of images of the object acquired by a monocular camera of fixed focal setting from different viewpoints. Object features are tracked over a range of distances from the camera with a small depth of field, leading to a varying degree of defocus for each feature. Information on absolute depth is obtained based on a Depth-from-Defocus approach. The parameters of the point spread functions estimated by Depth-from-Defocus are used as a regularisation term for Structure-from-Motion. The reprojection error obtained from Bundle Adjustment and the absolute depth error obtained from Depth-from-Defocus are simultaneously minimised for all tracked object features. The proposed method yields absolutely scaled 3D coordinates of the scene points without any prior knowledge about the structure of the scene. Evaluating the algorithm on real-world data we demonstrate that it yields typical relative errors between 2 and 3 percent. Possible applications of our approach are self-localisation and mapping for mobile robotic systems and pose estimation in industrial machine vision.

Depth perception is crucial for many applications including robotics, UAV and autonomous driving. The visual sense, as well as cameras, map the 3D world on a 2D representation, losing the dimension representing depth. A way to recover 3D information from 2D images is to record and join data from multiple viewpoints. In case of a stereo setup, 4D data is gained. Existing methods to recover 3D information are computationally expensive. We propose a new, more intuitive method to recover 3D objects out of event-based stereo data, by using a Self-Organizing Map to solve the correspondence problem and establish a structure similar to a voxel grid. Our approach, as it is also computationally expensive, copes with performance issues by massive parallelization. Furthermore, the relatively small voxel grid makes this a memory friendly solution. This technique is very powerful as it does not need any prior knowledge of extrinsic and intrinsic camera parameters. Instead, those parameters and also the lens distortion are learned implicitly. Not only do we not require a parallel camera setup, as many existing methods, we do not even need any information about the alignment at all. We evaluated our method in a qualitative analysis and finding image correspondences.