VDN: Variant‐depth network for motion deblurring

Facial editing is an important task in vision and graphics with numerous applications. However, existing works are incapable to deliver a continuous and fine-grained editing mode (e.g., editing a slightly smiling face to a big laughing one) with natural interactions with users. In this work, we propose Talk-to-Edit, an interactive facial editing framework that performs fine-grained attribute manipulation through dialog between the user and the system. Our key insight is to model a continual "semantic field" in the GAN latent space. 1) Unlike previous works that regard the editing as traversing straight lines in the latent space, here the fine-grained editing is formulated as finding a curving trajectory that respects fine-grained attribute landscape on the semantic field. 2) The curvature at each step is location-specific and determined by the input image as well as the users' language requests. 3) To engage the users in a meaningful dialog, our system generates language feedback by considering both the user request and the current state of the semantic field. We also contribute CelebA-Dialog, a visual-language facial editing dataset to facilitate large-scale study. Specifically, each image has manually annotated fine-grained attribute annotations as well as template-based textual descriptions in natural language. Extensive quantitative and qualitative experiments demonstrate the superiority of our framework in terms of 1) the smoothness of fine-grained editing, 2) the identity/attribute preservation, and 3) the visual photorealism and dialog fluency. Notably, user study validates that our overall system is consistently favored by around 80% of the participants. Our project page is https://www.mmlab-ntu.com/project/talkedit/.

Image quality is relevant to the performance of computer vision applications. The interference of rain streaks often greatly depreciates the visual effect of images. It is a traditional and critical vision challenge to remove rain streaks from rainy images. In this paper, we introduce a deep connectionist screen blend model for single-image rain removal research. The novel deep structure is mainly composed of shortcut connections, and ends with sibling branches. The specific architecture is designed for joint optimization of heterogeneous but related tasks. In particular, a feature-level task is design to preserve object edges which tend to be lost in de-rained images. Moreover, a comprehensive image quality assessment is an additional vision task for further improvement on de-rained results. Instead of using rules of thumb, we propose an actionable method to dynamically assign appropriate weighting coefficients for all vision tasks we use. On the other hand, various factors such as haze also give rise to weak visual appeal of rainy images. To remove these adverse factors, we develop an image enhancement framework which enables the hyperparameters to be optimized in an adaptive way, and efficiently improves the perceived quality of de-rained results. The effectiveness of the proposed de-raining system has been verified by extensive experiments, and most results of our method are impressive. The source code and more de-rained results will be available online.

In this paper, we propose a simple yet universal network termed SeqTR for visual grounding tasks, e.g., phrase localization, referring expression comprehension (REC) and segmentation (RES). The canonical paradigms for visual grounding often require substantial expertise in designing network architectures and loss functions, making them hard to generalize across tasks. To simplify and unify the modeling, we cast visual grounding as a point prediction problem conditioned on image and text inputs, where either the bounding box or binary mask is represented as a sequence of discrete coordinate tokens. Under this paradigm, visual grounding tasks are unified in our SeqTR network without task-specific branches or heads, e.g., the convolutional mask decoder for RES, which greatly reduces the complexity of multi-task modeling. In addition, SeqTR also shares the same optimization objective for all tasks with a simple cross-entropy loss, further reducing the complexity of deploying hand-crafted loss functions. Experiments on five benchmark datasets demonstrate that the proposed SeqTR outperforms (or is on par with) the existing state-of-the-arts, proving that a simple yet universal approach for visual grounding is indeed feasible. Source code is available at https://github.com/sean-zhuh/SeqTR.

Abstract4D human sensing and modeling are fundamental tasks in vision and graphics with numerous applications. With the advances of new sensors and algorithms, there is an increasing demand for more versatile datasets. In this work, we contribute HuMMan, a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames. HuMMan has several appealing properties: 1) multi-modal data and annotations including color images, point clouds, keypoints, SMPL parameters, and textured meshes; 2) popular mobile device is included in the sensor suite; 3) a set of 500 actions, designed to cover fundamental movements; 4) multiple tasks such as action recognition, pose estimation, parametric human recovery, and textured mesh reconstruction are supported and evaluated. Extensive experiments on HuMMan voice the need for further study on challenges such as fine-grained action recognition, dynamic human mesh reconstruction, point cloud-based parametric human recovery, and cross-device domain gaps (Homepage: https://caizhongang.github.io/projects/HuMMan/).

Unconditional human image generation is an important task in vision and graphics, enabling various applications in the creative industry. Existing studies in this field mainly focus on “network engineering” such as designing new components and objective functions. This work takes a data-centric perspective and investigates multiple critical aspects in “data engineering”, which we believe would complement the current practice. To facilitate a comprehensive study, we collect and annotate a large-scale human image dataset with over 230K samples capturing diverse poses and textures. Equipped with this large dataset, we rigorously investigate three essential factors in data engineering for StyleGAN-based human generation, namely data size, data distribution, and data alignment. Extensive experiments reveal several valuable observations w.r.t. these aspects: 1) Large-scale data, more than 40K images, are needed to train a high-fidelity unconditional human generation model with a vanilla StyleGAN. 2) A balanced training set helps improve the generation quality with rare face poses compared to the long-tailed counterpart, whereas simply balancing the clothing texture distribution does not effectively bring an improvement. 3) Human GAN models that employ body centers for alignment outperform models trained using face centers or pelvis points as alignment anchors. In addition, a model zoo and human editing applications are demonstrated to facilitate future research in the community. Code and models are publicly available (Project page: https://stylegan-human.github.io/. Code and models: https://github.com/stylegan-human/StyleGAN-Human.)KeywordsHuman image generationData-centricStyleGAN

Visual saliency detection aims at finding regions of interest which contain relevant information in images. In the last years, several saliency methods have been proposed, however, it is still a challenging task in visualization, graphics and computer vision. Visual saliency has been useful in many tasks such as object segmentation, object detection, image retrieval, place recognition, human-computer interaction, among others. In this work, we present the design of a Genetic Programming Framework to improve the saliency maps generated from a determined saliency method. As output, we obtain a sequence of operators to improve a saliency map. We have tested this approach by using three saliency methods of the state-of-the-art. The validation of the generated solutions have been tested in three visual saliency image datasets. The results of the experiments show that the solution found by Genetic Programming outperforms the original input saliency model.

The paper describes a hierarchical color clustering approach for the segmentation of textured images. It is a bottom-up type split and merge operation performed in two steps: first, the operation is carried out using the K-means algorithm in the local windows of an input image to capture the local textural primitives. This is performed in the (R,G,B)-color space. This results in two color classes per local window. Second, the merge operation is employed using the same K-means algorithm module to group the pattern classes resulting from the split operation. This forms the global boundaries of the texture fields present in the input scene, The proposed algorithm is also suitable for a special purpose VLSI chip implementation.

The fast hybrid operator splitting (HOS) and stable uniformization (UNI) methods have been proposed to save computation cost and enhance stability for Markov chain model in cardiac cell simulations. Moreover, Chen-Chen-Luo's quadratic adaptive algorithm (CCL) combined with HOS or UNI was used to improve the tradeoff between speedup and stability, but without considering accuracy. To compromise among stability, acceleration, and accuracy, we propose a generalized Trotter operator splitting (GTOS) method combined with CCL independent of the asymptotic property of a particular ion-channel model. Due to the accuracy underestimation of the mixed root mean square error (MRMSE) method, threshold root mean square error (TRMSE) is proposed to evaluate computation accuracy. With the fixed time-step RK4 as a reference, the second-order GTOS combined with CCL (30.8-fold speedup) for the wild-type Markov chain model with nine states (WT-9 model) or (7.4-fold) for the wild-type Markov chain model with eight states (WT-8 model) is faster than UNI combined with CCL (15.6-fold) for WT-9 model or (1.2-fold) for WT-8 model, separately. Besides, the second-order GTOS combined with CCL has 3.81% TRMSE for WT-9 model or 4.32% TRMSE for WT-8 model more accurate than 72.43% TRMSE for WT-9 model or 136.17% TRMSE for WT-8 model of HOS combined with CCL. To compromise speedup and accuracy, low-order GTOS combined with CCL is suggested to have the advantages of high precision and low computation cost. For high-accuracy requirements, high-order GTOS combined with CCL is recommended. Graphical abstract.

As the main observed illuminant outdoors, the sky is a rich source of information about the scene. However, it is yet to be fully explored in computer vision because its appearance in an image depends on the sun position, weather conditions, photometric and geometric parameters of the camera, and the location of capture. In this paper, we analyze two sources of information available within the visible portion of the sky region: the sun position, and the sky appearance. By fitting a model of the predicted sun position to an image sequence, we show how to extract camera parameters such as the focal length, and the zenith and azimuth angles. Similarly, we show how we can extract the same parameters by fitting a physically-based sky model to the sky appearance. In short, the sun and the sky serve as geometric calibration targets, which can be used to annotate a large database of image sequences. We test our methods on a high-quality image sequence with known camera parameters, and obtain errors of less that 1% for the focal length, 1° for azimuth angle and 3° for zenith angle. We also use our methods to calibrate 22 real, low-quality webcam sequences scattered throughout the continental US, and show deviations below 4% for focal length, and 3° for the zenith and azimuth angles. Finally, we demonstrate that by combining the information available within the sun position and the sky appearance, we can also estimate the camera geolocation, as well as its geometric parameters. Our method achieves a mean localization error of 110 km on real, low-quality Internet webcams. The estimated viewing and illumination geometry of the scene can be useful for a variety of vision and graphics tasks such as relighting, appearance analysis and scene recovery.

Inferring the scene illumination from a single image is an essential yet challenging task in computer vision and computer graphics. Existing works estimate lighting by regressing representative illumination parameters or generating illumination maps directly. However, these methods often suffer from poor accuracy and generalization. This paper presents Geometric Mover's Light (GMLight), a lighting estimation framework that employs a regression network and a generative projector for effective illumination estimation. We parameterize illumination scenes in terms of the geometric light distribution, light intensity, ambient term, and auxiliary depth, which can be estimated by a regression network. Inspired by the earth mover's distance, we design a novel geometric mover's loss to guide the accurate regression of light distribution parameters. With the estimated light parameters, the generative projector synthesizes panoramic illumination maps with realistic appearance and high-frequency details. Extensive experiments show that GMLight achieves accurate illumination estimation and superior fidelity in relighting for 3D object insertion. The codes are available at https://github.com/fnzhan/Illumination-Estimation.

3D face models have been widely applied in various fields (e.g. biometrics, movies, video games). Especially, it is one of the most popular and challenging tasks in computer vision and computer graphics to reconstruct a 3D face model only with single camera without attaching landmarks and projecting laser dots or structured light patterns on a face. For example, Maejima proposed a method, based on generic-model approach, which can quickly reconstruct a 3D face model from 2D single photograph using a deformable face model [Maejima et al. 2008]. However, since it supposes input as a frontal face image, this method cannot express the individual facial parts' geometry such as height of nose and cheek contour accurately.

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for large-range exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) We introduce a new panorama dataset Upright360 , comprising 9k high-quality and upright panorama images, and finetune the advanced Flux model on Upright360 for high-quality, upright and consistent panorama generation related tasks. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications. More examples please visit our webpage: ys-imtech.github.io/projects/LayerPano3D/.

Point cloud registration is a basic task in computer vision and computer graphics. Recently, deep learning-based end-to-end methods have made great progress in this field. One of the challenges of these methods is to deal with partial-to-partial registration tasks. In this work, we propose a novel end-to-end framework called MCLNet that makes full use of multi-level consistency for point cloud registration. First, the point-level consistency is exploited to prune points located outside overlapping regions. Second, we propose a multi-scale attention module to perform consistency learning at the correspondence-level for obtaining reliable correspondences. To further improve the accuracy of our method, we propose a novel scheme to estimate the transformation based on geometric consistency between correspondences. Compared to baseline methods, experimental results show that our method performs well on smaller-scale data, especially with exact matches. The reference time and memory footprint of our method are relatively balanced, which is more beneficial for practical applications.

The goal of infrared and visible image fusion (IVIF) is to combine complementary information from the source images. However, due to the absence of ground truth, most fusion algorithms just utilize information from source images, which cannot provide targeted guidance for network learning, leading to suboptimal outcomes. Moreover, numerous methods focus solely on modifying network architecture to improve fusion performance, without optimizing fusion algorithms from other perspectives. To tackle these problems, we propose a real-time end-to-end IVIF network based on an adaptive guidance module (AGMFusion). On the one hand, by combining the state-of-the-art (SOTA) methods and best fusion outcomes during the training process, the adaptive guidance module (AGM) effectively facilitates collaborative guidance for network training. Additionally, based on the AGM, we devise a loss function that includes content loss and guidance loss. We further balance these two loss components with the adaptive weight to boost the performance of our framework. On the other hand, AGMFusion is a lightweight image fusion framework that can generate highly perceptual fused images while maintaining excellent real-time performance, which can be potentially deployed as a preprocessing unit for various vision tasks. According to extensive comparative and generalization experiments, AGMFusion exceeds existing methods in visual appeal and quantitative metrics. Importantly, performance comparisons of numerous frameworks in running efficiency and object detection highlight the advantages of our approach. The source code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/liushh39/AGMFusion</uri>.

Image restoration is a challenging ill-posed problem which also has been a long-standing issue. In the past few years, the convolution neural networks (CNNs) almost dominated the computer vision and had achieved considerable success in different levels of vision tasks including image restoration. However, recently the Swin Transformer-based model also shows impressive performance, even surpasses the CNN-based methods to become the state-of-the-art on high-level vision tasks. In this paper, we proposed a restoration model called SUNet which uses the Swin Transformer layer as our basic block and then is applied to UNet architecture for image denoising. The source code and pre-trained models are available at https://github.com/FanChiMao/SUNet.