Improving the Color Accuracy of Lighting Estimation Models

High dynamic range (HDR) imaging is an indispensable technique in modern photography. Traditional methods focus on HDR reconstruction from multiple images, solving the core problems of image alignment, fusion, and tone mapping, yet having a perfect solution due to ghosting and other visual artifacts in the reconstruction. Recent attempts at single-image HDR reconstruction show a promising alternative: by learning to map pixel values to their irradiance using a neural network, one can bypass the align-and-merge pipeline completely yet still obtain a high-quality HDR image. In this work, we propose a weakly supervised learning method that inverts the physical image formation process for HDR reconstruction via learning to generate multiple exposures from a single image. Our neural network can invert the camera response to reconstruct pixel irradiance before synthesizing multiple exposures and hallucinating details in under- and over-exposed regions from a single input image. To train the network, we propose a representation loss, a reconstruction loss, and a perceptual loss applied on pairs of under- and over-exposure images and thus do not require HDR images for training. Our experiments show that our proposed model can effectively reconstruct HDR images. Our qualitative and quantitative results show that our method achieves state-of-the-art performance on the DrTMO dataset. Our code is available at https://github.com/VinAIResearch/single_image_hdr.

In this paper, we present a novel high dynamic range (HDR) imaging method using a single input image. Conventional multiple image-based HDR methods are successful only on condition that there is no motion in the scene during the acquisition of multiple, differently exposed low dynamic range (LDR) images. If these constraints are not satisfied, a ghost artifact is produced in the resulting HDR image. In order to overcome these limitations, we generate multiple, differently exposed LDR images from a single input image. We call these multiple images a set of layered exposed (LE) images. In order to generate an appropriate set of LE images, the proposed method divides input image into 9 subregions and computes local mean in each subregion to estimate the minimum and maximum local mean. The estimated local means become ranges of histogram equalization (HE). HDR image is generated by fusing differently exposed LE images. More specifically, given a set of LE images, we perform weighted fusion to produce the resulting HDR image, which is inherently free from ghost artifacts since all LE images are geometrically identical. Experimental results show that the proposed method outperforms the existing algorithms in the sense of both removing ghost artifacts and enhancement of image contrast.

Most current three-dimensional reconstruction methods often require at least two or more input images to reconstruct the shape of the object. Traditionally, it is hard to reconstruct the 3D shape only from one single input image. In this paper, we propose an effective method to reconstruct the 3D texture surface from a single input image for the texture with similar appearances and reflectance properties. Exactly, the patch matching and optimization methods have been combined to resolve the under-constrained problem of 3D reconstruction from one single image. Experimental results have verified the effectiveness of the proposed method in 3D reconstruction according to realistic perception.

This paper proposes a method for discovering the primary objects in single images by learning from videos in a purely unsupervised manner—the learning process is based on videos, but the generated network is able to discover objects from a single input image. The rough idea is that an image typically consists of multiple object instances (like the foreground and background) that have spatial transformations across video frames and they can be sparsely represented. By exploring the sparsity representation of a video with a neural network, one may learn the features of each object instance without any labels, which can be used to discover, recognize, or distinguish object instances from a single image. In this paper, we consider a relatively simple scenario, where each image roughly consists of a foreground and a background. Our proposed method is based on encoder-decoder structures to sparsely represent the foreground, background, and segmentation mask, which further reconstruct the original images. We apply the feed-forward network trained from videos for object discovery in single images, which is different from the previous co-segmentation methods that require videos or collections of images as the input for inference. The experimental results on various object segmentation benchmarks demonstrate that the proposed method extracts primary objects accurately and robustly, which suggests that unsupervised image learning tasks can benefit from the sparsity of images and the inter-frame structure of videos.

Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handling both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learned in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illumination. This work is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide extensive evaluation and ablative studies of our model and also provide an application, where any face can be relighted using textual input.

Image degraded by haze is a critical aspect in today's environment while getting a high-quality haze-free image remains an important task in computer vision. In recent year, many works have been done to improve the visibility of image taken under bad weather. Conventional designs use multiple image and/ or single image to deal with haze removal. In this paper, we use a simple but effective prior, a variation of distance (VoD) prior, to estimate the transmission map and remove haze from a single input image. The VoD prior is developed based on the idea that the outdoor visibility of images taken under hazy weather conditions seriously reduced when the distance increases. The thickness of the haze can be estimated effectively and a haze-free image can be recovered by adopting the VoD prior and the new haze imaging model. Our method is stable to image local regions containing objects in different depths. Our experiments showed that the proposed method achieved better results than several state-of-the-art methods, and it can be implemented very quickly. Our method due to its fast speed and the good visual effect is suitable for real-time applications. This work confirms that estimating the transmission map using the distance information instead the color information is a crucial point in image enhancement and especially single image haze removal.

Although neural radiance fields (NeRF) have shown impressive advances in novel view synthesis, most methods require multiple input images of the same scene with accurate camera poses. In this work, we seek to substantially reduce the inputs to a single unposed image. Existing approaches using local image features to reconstruct a 3D object often render blurry predictions at viewpoints distant from the source view. To address this, we propose to leverage both the global and local features to form an expressive 3D representation. The global features are learned from a vision transformer, while the local features are extracted from a 2D convolutional network. To synthesize a novel view, we train a multi-layer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. This novel 3D representation allows the network to reconstruct unseen regions without enforcing constraints like symmetry or canonical coordinate systems. Our method renders novel views from just a single input image, and generalizes across multiple object categories using a single model. Quantitative and qualitative evaluations demonstrate that the proposed method achieves state-of-the-art performance and renders richer details than existing approaches. https://cseweb.ucsd.edu/ %7eviscomp/projects/VisionNeRF/

We propose a novel framework to reconstruct super-resolution human shape from a single low-resolution input image. The approach overcomes limitations of existing approaches that reconstruct 3D human shape from a single image, which require high-resolution images together with auxiliary data such as surface normal or a parametric model to reconstruct high-detail shape. The proposed framework represents the reconstructed shape with a high-detail implicit function. Analogous to the objective of 2D image super-resolution, the approach learns the mapping from a low-resolution shape to its high-resolution counterpart and it is applied to reconstruct 3D shape detail from low-resolution images. The approach is trained end-to-end employing a novel loss function which estimates the information lost between a low and high-resolution representation of the same 3D surface shape. Evaluation for single image reconstruction of clothed people demonstrates that our method achieves high-detail surface reconstruction from low-resolution images without auxiliary data. Extensive experiments show that the proposed approach can estimate super-resolution human geometries with a significantly higher level of detail than that obtained with previous approaches when applied to low-resolution images. https://marcopesavento.github.io/SuRS/.

Despite the progress of 3D Gaussian Splatting (3DGS), reconstructing one-shot animatable 3D human avatars from a single image remains a challenging task. Existing 3DGS-based methods primarily rely on appearance observation and motion cues from monocular videos to reconstruct animatable 3D avatars. However, when applied to single-image setups, these methods struggle to extract accurate 3D features from a 2D image, limiting their ability to capture fine-grained appearance details and dynamic deformation, especially from challenging perspectives. In this work, we propose SIAvatar, a novel single-image human reconstruction method that integrates diffusion-based appearance prior and parametric human model geometry prior within a 3DGS framework. Secondly, a vertex-based adaptive Gaussian densification scheme is introduced to effectively represent human geometry while mitigating artifacts. Extensive experiments demonstrate that SIAvatar generates realistic 3D avatars with plausible appearance details and novel pose animation from a single input image. Project page: https://siavatar.github.io/.

A technique for extracting intrinsic images, including the reflectance and illumination images, from a single color image is presented. The technique first convolves the input image with a prescribed set of derivative filters. The pixels of filtered images are then classified into reflectance-related or illumination-related based on a set of chromatic characteristics of pixels calculated from the input image. Chromatic characteristics of pixels are defined by a photometric reflectance model based on the Kubelka-Munk color theory. From the classification results of the filtered images, the intrinsic images of the input image can be computed. Real images have been utilized in our experiments. The results have indicated that the proposed technique can effectively extract the intrinsic images from a single image.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Given a single chair image, could we wake it up by reconstructing its 3D shape and skeleton, as well as animating its plausible articulations and motions, similar to that of human modeling? It is a new problem that not only goes beyond image-based object reconstruction but also involves articulated animation of generic objects in 3D, which could give rise to numerous downstream augmented and virtual reality applications. In this paper, we propose an automated approach to tackle the entire process of reconstruct such generic 3D objects, rigging and animation, all from single images. two-stage pipeline has thus been proposed, which specifically contains a multi-head structure to utilize the deep implicit functions for skeleton prediction. Two in-house 3D datasets of generic objects with high-fidelity rendering and annotated skeletons have also been constructed. Empirically our approach demonstrated promising results; when evaluted on the related sub-tasks of 3D reconstruction and skeleton prediction, our results surpass those of the state-of-the-arts by a noticeable margin. Our code and datasets are made publicly available at the dedicated project website.KeywordsObject reconstructionObject rigging

There is a growing interest for the broad use of Augmented Reality (AR) and Virtual Reality (VR) in the fields of bioinformatics and cheminformatics to visualize complex biological and chemical structures. AR and VR technologies allow for stunning and immersive experiences, offering untapped opportunities for both research and education purposes. However, preparing 3D models ready to use for AR and VR is time-consuming and requires a technical expertise that severely limits the development of new contents of potential interest for structural biologists, medicinal chemists, molecular modellers and teachers. Herein we present the RealityConvert software tool and associated website, which allow users to easily convert molecular objects to high quality 3D models directly compatible for AR and VR applications. For chemical structures, in addition to the 3D model generation, RealityConvert also generates image trackers, useful to universally call and anchor that particular 3D model when used in AR applications. The ultimate goal of RealityConvert is to facilitate and boost the development and accessibility of AR and VR contents for bioinformatics and cheminformatics applications. http://www.realityconvert.com. dfourch@ncsu.edu. Supplementary data are available at Bioinformatics online.

Augmented reality technologies undertake focus role which provide opportunity to harmonize the earthilyreality with virtual field and evaluate their integral elements.The very first examples of augmented realityapplications are viewers on the head,simulators,wearable tools,pocket PCs,desktop PCs and cameras whichare integrated externally but augmented reality applications have gain a great acceleration by developingtechnologies such as internet,becoming popular for the mobile devices and their mobile applicationsaccordingly and especially by the appearance of the smart wears and increase on their functions.Augmentedreality applications that are often used in defence,industry,medicine,trade and entertainment are now startedto be used in the education.Augmented reality applications achieve deep learning of the systems or objectswhich are hard to learn by attracting attention and cares of the students by providing their different viewsduring teaching and learning process.Also these applications supports students to use their creativity byexpanding their imagination.Especially if we consider today’s students are digital natives(Y and Z generation) who grow up in the multimedia world and inside the technology,it is a reality that augmented reality applications which have rich content in the learning-teaching environments can have a role that increase themotivations of the students instead of a classic course book. In the light of these information, in this study theresearch and application samples of augmented reality which is now used widely in teaching and learningprocess the effect of augmented reality on teaching and learning is handled.Keywords: Augmented reality, augmented reality applications in education

High dynamic range (HDR) imaging can represent a great range of real-world luminosity. In contrast, the traditional low dynamic range (LDR) imaging fails to represent a wide range of luminance since most digital cameras can capture a limited range of light intensity in a natural scene. Recent advances in deep learning allow reconstructing an HDR image from a single LDR image and surpass conventional methods performance. In this work, we propose a novel CNN for HDR image reconstruction based on residual learning and attention mechanism. The proposed network adopts an autoencoder structure with residual blocks trained in a fully end-to-end manner. Residual learning boosts the performance by optimizing the network to converge faster. Moreover, the attention mechanism allows the network to select and enhance meaningful features that will contribute to the reconstruction of the HDR image. In addition, we employ a contextual attention module to perform patch replacement on deep feature maps to help recover information in over-exposed areas. Extensive quantitative and qualitative experiments on public HDR datasets demonstrate the ability of our proposed method to effectively reconstruct a visually pleasing HDR image from a single LDR image and outperform existing approaches.