Hybrid method for multiple viewpoint projection holography

Efficiency and accuracy of semi-global matching (SGM) make it outperform many stereo matching algorithms and is widely used under challenging occasions. However, SGM only incorporates information along a scanline in each pass and lacks interaction between scanlines, resulting in streak artifacts in the disparity image. We introduce a local edge-aware filtering method to SGM to enhance the interaction of neighboring scanlines, since streak artifacts can be avoided. We use bilateral weights based on intensity similarity and spatial affinity between pixels to build connections among scanlines. In each pass, we recursively estimate the aggregated cost of SGM and compute the weighted average of aggregated costs for pixels in the orthogonal direction to obtain the output of our method along each scanline. As one-dimensional bilateral filtering is used in our method, the extra computation is linear to image resolution and label space, which is a small fraction of that needed by SGM. We present ablation studies using stereo pairs under both constrained and natural conditions to verify the effectiveness of our method. Extensive experiments on Middlebury and Karlsruhe Institute of Technology and Toyota Technology Institute datasets demonstrate that our method removes all streak artifacts, improves the quality of the disparity image, and outperforms many other non-local cost aggregation approaches.

We propose a novel lightweight network for stereo estimation. Our network consists of a fully-convolutional densely connected neural network (FC-DCNN) that computes matching costs between rectified image pairs. Our FC-DCNN method learns expressive features and performs some simple but effective postprocessing steps. The densely connected layer structure connects the output of each layer to the input of each subsequent layer. This network structure and the fact that we do not use any fully-connected layers or 3D convolutions leads to a very lightweight network. The output of this network is used in order to calculate matching costs and create a cost-volume. Instead of using time and memory-inefficient cost-aggregation methods such as semiglobal matching or conditional random fields in order to improve the result, we rely on filtering techniques, namely median filter and guided filter. By computing a left-right consistency check we get rid of inconsistent values. Afterwards we use a watershed foreground-background segmentation on the disparity image with removed inconsistencies. This mask is then used to refine the final prediction. We show that our method works well for both challenging indoor and outdoor scenes by evaluating it on the Middlebury, KITTI and ETH3D benchmarks respectively. Our full framework is available at https://github.com/thedodo/FC-DCNN

Introduction/purposeDigital subtraction angiography (DSA) remains the gold standard for the evaluation of extra- and intracranial vascular pathology. Typically, acquisition of multiple angiographic projections is needed to either elongate the vessel(s)...

In this paper, a virtual optical encryption open source software system for image security is proposed. Our system makes it possible to encrypt and decrypt images based on the optical theorem, which includes the virtual optical image encryption technique (VOIET), the data hiding technique (DHT), the data extraction technique (DET), and the virtual optical image decryption technique (VOIDT). The optical theorem is based the digitized double-random-phase algorithm, which is called the virtual optical theorem. In the system, secret data can be embedded into the encrypted image data without distortion. The comparison of data hiding capacity and reconstructed image quality is performed in the experiment. Our system is designed and implemented by the open source software, which is called the Virtual Optical Image Encryption System (VOIES) and released on the Open Source Software Foundry (OSSF).

Depth estimation has vital importance in various applications like Advance driver assistance systems, Robot navigation, 3D movies, etc. Depth information is represented using disparity maps which are generated from various stereo correspondence algorithms. This paper proposes an implementation of Semi global block matching algorithm on raspberry pi using python. The proposed algorithm generates a disparity map by block wise matching and smoothness constraint. The novel algorithm is compared with other algorithms on the standard dataset. It provides an efficient accuracy over local methods. The real time implementation of SGM algorithm on the system uses a block size of 21 × 21 for images having resolution 1280×720P. The system provides faster processing time for the proposed algorithm.

The mixed reality makes the user's realism strong or weak, depending on the fusion effect of the virtual object and the real scene, and the key factor affecting the virtual-real fusion effect is whether the virtual and real object in the synthetic scene has occlusion consistency. In this paper, a method is proposed that utilizes color and depth information provided by an RGB-D camera to improved occlusion. In this method, based on the GPU, extracting the ROI area where the virtual object is located (the area where the virtual-real occlusion occurs),then introducing the real scence color image as the guide image to modified joint bilateral filtering the depth image to repair the wrong edge information and "black hole" of the depth image;pixel-by-pixel point to determine the depth relationship between the virtual object and real object, to achieve virtual-real occlusion rendering blend;for the still existing "sawtooth artifacts" using delay-coloring only fuzzy local boundary algorithm, to achieve accurate and smoothing edges virtual-real occlusion to enhance visual effects.

In this paper, we propose a method to improve ego-motion estimation. In the proposed method, feature points from input binocular stereo image pairs are detected using the Speeded Up Robust Features method and descriptor vectors are generated for the features. Semi-Global Matching is then performed to ensure the quality and quantity of the matching features, which are used in conjunctions with camera parameters to calculate the 3D coordinates for the image features. Kalman filter and adaptive matching are the used to perform feature tracking. The calculated 3D coordinates, as well as the 2D projections and camera parameters are used in Efficient Perspective-n-Point Camera Pose Estimation to find the camera motions. Finally, Sparse Bundle Adjustment is used to perform optimization of the estimated motions.

Automatic building detection from very-high-resolution (VHR) satellite images is a difficult task. The detection accuracy is usually limited by spectral ambiguities and the uncertainties of the available height information. Feature extraction and training sampling collection for supervised methods are other sources of uncertainty. Most widely used VHR sensors have shorter revisit cycles (IKONOS/GeoEye-1/2, 3 days; WorldView 1/2, 1.1 days) due to large off-nadir viewing angles and hence are able to perform consistent acquisition of mono or stereo images. In this article, we investigate the possibility of using high-temporal stereo VHR images to enhance remote-sensing image interpretation under the context of building detection. Digital surface models, which contain the height information, are generated for each date using semi-global matching. Pre-classification is performed combining the height and spectral information to obtain an initial building probability map. With a reference land cover map available for one date, the training samples of the other dates are automatically derived using a rule-based validating procedure. A spatiotemporal inference filter is developed considering the spectral, spatial, and temporal aspects to enhance the building probability maps. This aims at homogenizing the building probability values of spectrally similar pixels in the spatial domain and geometrically similar pixels in the temporal domain, while being robust to the silhouette of the images and geometric discrepancies of the multitemporal data. The effectiveness and robustness of the proposed method are evaluated by performing three experiments on six stereo pairs of the same region over a time period of five years (2006–2011). The area under curve (AUC) of the receiver operating characteristic and kappa statistic (κ) are employed to assess the results. These experiments show that spatiotemporal inference filtering largely improves the accuracy of the building probability map (average AUC = 0.95) while facilitating building extraction in snow-covered images. The resulting building probability maps can be further used for other applications (e.g. building footprint updating).

In this paper, we present a new and efficient multi-modal 3D-3D vascular registration algorithm, which transforms the 3D-3D registration problem into a multiple 2D-3D vascular registration problem. Along each orthogonal axis, projected 2D image from a segmented binary 3D floating volume is compared with maximum intensity projection (MIP) image of the reference volume. At the preprocessing stage of the floating image volume, vessels are segmented and represented by a number of spheres with centers located at the skeleton points of the vessels and radii equal to the distance from the skeleton points to their closest boundary. To generate projected images from the binary 3D volume, instead of using the conventional ray-casting technique, the spheres are projected to the three orthogonal projection planes. The discrepancy between the projected image and the reference MIP image is measured by a relatively simple similarity measure, sum of squared differences (SSD). By visual comparison, we found that the performances of our method and the Mutual Information (MI)-based method are visually comparable. Moreover, based on the experimental results, our method for 3D-3D vascular registration is more computationally efficient than the MI-based method.

Currently, ray tracing and image-based lighting (IBL) have shortcomings when rendering the metallic virtual object displayed in the holographic pyramid in mixed reality. Ray tracing can hardly achieve the interactive frame rates, and IBL cannot accurately render the reflection result of the foreground near the virtual object. In this paper, we propose a mixed reality rendering method to render glossy and specular reflection effects on metallic virtual objects displayed in the holographic pyramid based on the surrounding real environment at interactive frame rates. First, we acquire the real environment data with four RGBD cameras and a panoramic camera; then, we introduce a foreground point cloud generation method to extract a temporally stable foreground point cloud from RGBD videos captured in real time; after this, we propose an efficient ray tracing method to render the dynamic glossy and specular reflections on the virtual objects that are displayed in the holographic pyramid. We test our method on several real and synthetic scenes. Compared with IBL and screen-space ray tracing, our method can generate the rendering results closer to the ground truth at the same time cost. Compared with Monte Carlo path tracing, our method is 2.5-4.5× faster in generating rendering results of the comparable quality.

For urban planning purposes updated information on land-use and urban structures and change dynamics are essential. Satellite Earth observation is a very effective information source, to provide actual representations of the situation ‘as is’ in regular time steps. The Pleiades constellation offers very high resolution imagery with a triple-stereo acquisition. The intrinsic combination of thematic and height information is an additional asset provided by recent VHR stereo imagery. Digital surface models (DSM) can be derived through stereo matching algorithms, one of which is semi-global matching (SGM). SGM integrates local and global matching methods with a short processing time and accurate surface generation. We use the generated DSM for extracting 3-D features which enrich object-based class models of composite urban objects. These multi-purpose 3-D object feature datasets can be used for various planning and monitoring purposes, revealing height-related characteristics of urban objects (blocks, neighbourhoods, parks, industrial sites, etc.). In addition object-wise aggregated height information can be used to better assess the adherence of actual building activities to zoning or land development plans, e.g. with respect to maximum or average height in recreational areas etc. As compared to detailed and potentially more accurate DSMs generated from LiDAR data, a (3-)stereo DSM has the advantage of recording height and spectral information instantaneously, so not to lose any information in high dynamic urban settings due to a time gap. Height and colour information, when integrated by spatial context and multi-scale OBIA methods, enable a wealth of new applications in urban feature characterisation and 3D city model applications.

Instrumented with a single depth camera, a stereoscopic projector, and a curved screen, MirageTable is an interactive system designed to merge real and virtual worlds into a single spatially registered experience on top of a table. Our depth camera tracks the user's eyes and performs a real-time capture of both the shape and the appearance of any object placed in front of the camera (including user's body and hands). This real-time capture enables perspective stereoscopic 3D visualizations to a single user that account for deformations caused by physical objects on the table. In addition, the user can interact with virtual objects through physically-realistic freehand actions without any gloves, trackers, or instruments. We illustrate these unique capabilities through three application examples: virtual 3D model creation, interactive gaming with real and virtual objects, and a 3D teleconferencing experience that not only presents a 3D view of a remote person, but also a seamless 3D shared task space. We also evaluated the user's perception of projected 3D objects in our system, which confirmed that the users can correctly perceive such objects even when they are projected over different background colors and geometries (e.g., gaps, drops).

Endoscopic surgery causes less tissue injury compared to open surgical techniques, thus promoting more rapid recuperation and reduced post-operative pain. Endoscopy, however, allows the surgeon to visualise only the anatomical surface of the surgical site, with a relatively narrow field of view. Moreover, the 2D video captured by the conventional endoscope does not provide depth perception of the surgical scene. In this study, these limitations have been addressed with the development of an augmented reality (AR) system with stereoscopic visualisation. A phantom and its 3D CT model were used, respectively, to form the real and virtual parts of the AR. The virtual environment camera pose was tracked using algorithms for image feature detection, feature matching and Perspective-n-Point applied on the endoscopic image and the 3D virtual model-rendered image. The endoscope video frame- and the virtual model-rendered images were superimposed to form the AR composite view. The depth buffer (z-buffer) of the rendering window was further used to make a stereo pair of the AR image. The AR system produced a stereo composite view having well-aligned real and virtual components. The RMS error of the real and virtual image contours registration was 9.6 ± 6.7 mm. Correlation coefficients between the depth map from z-buffer and a depth camera was between 0.60 and 0.96 (p < 0.05). The AR system requires further improvement to be applicable at a higher frame rate of endoscope image acquisition. It also needs to include motion and deformation models when applied to animals or patients.

In my laboratory, the new localization method, Double Near-field Acoustic Holography (DNAH) method is being Developed. This method is developed to overcome the limit of acoustic holography methods, that the acoustic holography methods cannot measure low frequency noise precisely. In former presentations, the numerical simulation results, and basic experimental results are reported. In these presentations, it is verified that the DNAH method has good resolution for low frequency sound sources. But they are carried out for simple sound source(s), simulated point source(s). In this presentation, the experimental results for a commercial product will be presented. From these results, it is verified that the DNAH method has good resolution for low frequency noise of a commercial product between 10Hz and 200Hz, and the DNAH method has enough resolution for low frequency sound localization.

AI-based computer-aided diagnostic system of chest digital tomography synthesis: Demonstrating comparative advantage with X-ray-based AI systems