Joint Object Detection and Depth Estimation in Multiplexed Image

A cost-effective demosaicked image enhancement method for a single chip CMOS (complementary metal oxide semiconductor) image sensor which has on-chip ISP (image signal processor) is introduced. CMOS image sensors are becoming popular, because various circuits can be integrated with it into a single chip. In a single chip CMOS image sensor for handheld devices which requires low power consumption, ISP has to be implemented to small size while maintaining the high performance. Demosaicking and image enhancing are the largest blocks in the ISP, because they need some line memories. In this paper, we propose a new method to minimize the number of line memory. In the proposed method, contrast enhancement is done using the buffered data for demosaicking. The green channel is used instead of luminance. Experimental results indicate that the proposed approach enhances the image quality without additional line memories. The proposed method is implemented on FPGA chip in real time mode, and performed successfully.

This paper starts with the structural relationship of the head tracking system based on single imaging device, establishes equation set of projection relation, the rationality of the system is determined by analyzing the possibility of the equation solution, projection relationship is established, finally, the mathematical model of the two degree-of-freedom tracking system based on single image sensor is completed from theory.

Conventional stereo matching systems generate a depth map using two or more digital imaging sensors. It is difficult to use the small camera system because of their high costs and bulky sizes. In order to solve this problem, this paper presents a stereo matching system using a single image sensor with phase masks for the phase difference auto-focusing. A novel pattern of phase mask array is proposed to simultaneously acquire two pairs of stereo images. Furthermore, a noise-invariant depth map is generated from the raw format sensor output. The proposed method consists of four steps to compute the depth map: (i) acquisition of stereo images using the proposed mask array, (ii) variational segmentation using merging criteria to simplify the input image, (iii) disparity map generation using the hierarchical block matching for disparity measurement, and (iv) image matting to fill holes to generate the dense depth map. The proposed system can be used in small digital cameras without additional lenses or sensors.

Object localization and detection is the emerging area of computer vision and image processing that finds out the location of the object in the video frames or in the digital images. Object localization and detection have many challenges like occlusion, scale variations, intraclass similarities, illumination conditions, pose variations, etc. Multi-view object detection focuses on different views of objects like the top, bottom, side view, etc. We aim to discuss the different Object detection models based on Machine learning and Deep neural network approaches in a multi-view environment. We also discuss different datasets used for object detections and their applications. We apply some object detection models like SSD and Faster R-CNN to specific object categories of Open Image Dataset V6 and compare the results based on mAP (Mean Average Precision).KeywordsObject detectionConvolution neural networksR-CNNFast R-CNNFaster R-CNNEfficientDetCenterNet

In this paper, we propose MINE to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our approach is a continuous depth generalization of the Multiplane Images (MPI) by introducing the NEural radiance fields (NeRF). Given a single image as input, MINE predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our MINE outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Our source code is available at https://github.com/vincentfung13/MINE.

A compact off-axis three-mirror system, especially with a wide field of view and small f-number, continues to present challenges in the optical design field. To design a compact off-axis three-mirror system based on a freeform surface, an optimization method with a circular configuration and four parameters is proposed. In the proposed method, the f-number and angles of mirrors are combined to optimize and achieve high-quality imaging, which means the modulation transfer function is close to the diffraction limit. To evaluate the performance of the proposed method, four design examples with different fields of view (4∘×4∘, 6∘×6∘) and f-numbers (2, 1.75) were created. The results showed that the compact off-axis three-mirror system based on a freeform surface can be designed with a wide field of view, small f-number, and high-quality imaging using the proposed method. Unlike other design methods, the surface parameters of mirrors are obtained with only one calculation and are close to the final optimization results, which saves both time and resources for optimization. The findings indicate that the method is accurate and effective for designing compact off-axis three-mirror systems with freeform surfaces.

An implantable, remotely controlled medical device that incorporates an imaging/therapy ultrasound system may be minimally invasive and equipped with an anchoring portion for securing the device within a human body. Transducers for imaging/therapy may be manipulated remotely using motors and/or selectively actuated to obtain different fields of view and stereoscopic imaging. The implantable medical device can be in the shape of a disc, double disc, sphere or pellet, for example, and may be implanted during open surgery using a manipulatable tool or using a minimally invasive image-guided sheath or catheter. The imaging system comprises one or more ultrasound transducers and can be used to provide therapy to or obtain ultrasound images of a target and surrounding volumes or focal points. The device may be controlled and report data by wired or wireless means and, if wireless, permanently worn inside the body as the patient follows their normal daily routine.

We explore the role of top-down processes in the recovery of the 3D shape of faces from single images. Class familiarity appears to play a significant role in the perception of shape in single images, as is demonstrated by the perception of two-tone (“Mooney”) images. We prove that under standard shape from shading assumptions, a Mooney image can give rise to multiple different 3D reconstructions. We further present an algorithm that by using just a single reference shape model—the face of a different individual or an average face—can achieves veridical reconstructions from two-tone images as well as gray-level and color images downloaded from the Internet.KeywordsReference ModelEikonal EquationGray Level ImageReflectance FunctionLambertian SurfaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Synthesizing novel views from a single view image is a highly ill-posed problem. We discover an effective solution to reduce the learning ambiguity by expanding the single-view view synthesis problem to a multi-view setting. Specifically, we leverage the reliable and explicit stereo prior to generate a pseudo-stereo viewpoint, which serves as an auxiliary input to construct the 3D space. In this way, the challenging novel view synthesis process is decoupled into two simpler problems of stereo synthesis and 3D reconstruction. In order to synthesize a structurally correct and detail-preserved stereo image, we propose a self-rectified stereo synthesis to amend erroneous regions in an identify-rectify manner. Hard-to-train and incorrect warping samples are first discovered by two strategies, (1) pruning the network to reveal low-confident predictions; and (2) bidirectionally matching between stereo images to allow the discovery of improper mapping. These regions are then inpainted to form the final pseudo-stereo. With the aid of this extra input, a preferable 3D reconstruction can be easily obtained, and our method can work with arbitrary 3D representations. Extensive experiments show that our method outperforms state-of-the-art single-view view synthesis methods and stereo synthesis methods.

Most current three-dimensional reconstruction methods often require at least two or more input images to reconstruct the shape of the object. Traditionally, it is hard to reconstruct the 3D shape only from one single input image. In this paper, we propose an effective method to reconstruct the 3D texture surface from a single input image for the texture with similar appearances and reflectance properties. Exactly, the patch matching and optimization methods have been combined to resolve the under-constrained problem of 3D reconstruction from one single image. Experimental results have verified the effectiveness of the proposed method in 3D reconstruction according to realistic perception.

Visible and near-infrared image acquisition using a single conventional image sensor helps simpler implementation on mobile devices. This paper presents a signal separation method of visible and near-infrared images from an image which has mixed visible and near-infrared signals captured by a single CMOS sensor. In the separation process, the conditional generative adversarial network is applied to extract each visible and near-infrared image from the mixed one. For the experiment, a conventional digital camera is used with a simple modification. The experimental result shows that the proposed method well extracts both visible and near-infrared images.

In this paper, we proposed a three dimensional matching algorithm using geometrical invariants. Invariant relations between 3D objects and 2D images for object recognition has been already developed in Ref. [1]. We proposed a geometrical invariant approach for finding relation between 3D model and stereo image pair. Since the depth information is lost in a single 2D image, we cannot recognize an object perfectly. By constructing a 3D invariant space we can represent a 3D model as a set of points in the invariant space. While matching with the 2D image we can draw a set of invariant light rays in 3D, each ray passing through a 3D invariant model point. If enough rays intersect the model in 3D invariant space we can assume that the model is present in the image. But for a single image the method is not that much reliable as the depth information is never considered. In the proposed method, as the matching is performed using stereo image pair, it is more reliable and accurate.

Most of the available digital color cameras use a single image sensor with a color filter array (CFA) in acquiring an image. In order to produce a visible color image, a demosaicing process must be applied, which produces undesirable artifacts. An additional problem appears when the observed color image is also blurred. This paper addresses the problem of deconvolving color images observed with a single coupled charged device (CCD) from the super-resolution point of view. Utilizing the Bayesian paradigm, an estimate of the reconstructed image and the model parameters is generated. The proposed method is tested on real images.

Multi-instance Deep Learning of Ultrasound Imaging Data for Pattern Classification of Congenital Abnormalities of the Kidney and Urinary Tract in Children

This paper realizes a computational integral imaging reconstruction method via scale invariant feature transform (SIFT) and patch matching to improve the visual quality of reconstructed 3D view images. To our knowledge, the 3D view images reconstructed from the elemental images suffer from artifacts, which leads to degradations in the visual quality. To prevent image degradation, in this paper, we use the correct regions obtained from the view images taken directly from the original object or use patch matching to replace the distorted regions. However, the initial matching regions could not meet our requirements owing to the limitations of the equipment and the inevitable shortcomings of the experimental operation. To solve these problems, we adopt SIFT descriptors and perspective transform to get the satisfying correct regions. We present the simulation and experimental results of the 3D view images and the evaluation of the quality of the corresponding images to test the performance of the proposed method. The simulation and experimental results indicate that the proposed method can significantly improve the visual quality of the 3D view images and verify the feasibility and effectiveness of the proposed method.