Continuous Layout Editing of Single Images with Diffusion Models

Safety-critical applications, such as autonomous driving, require extensive multimodal data for rigorous testing. Methods based on synthetic data are gaining prominence due to the cost and complexity of gathering real-world data but require a high degree of realism and controllability in order to be useful. This paper introduces MObI, a novel framework for Multimodal Object Inpainting that leverages a diffusion model to create realistic and controllable object inpaintings across perceptual modalities, demonstrated for both camera and lidar simultaneously. Using a single reference RGB image, MObI enables objects to be seamlessly inserted into existing multimodal scenes at a 3D location specified by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, our 3D bounding box conditioning gives objects accurate spatial positioning and realistic scaling. As a result, our approach can be used to insert novel objects flexibly into multimodal scenes, providing significant advantages for testing perception models. Project page: https://alexbubu.com/mobi

Novel view synthesis from a single image has recently attracted a lot of attention, and it has been primarily advanced by 3D deep learning and rendering techniques. However, most work is still limited by synthesizing new views within relatively small camera motions. In this paper, we propose a novel approach to synthesize a consistent long-term video given a single scene image and a trajectory of large camera motions. Our approach utilizes an autoregressive Transformer to perform sequential modeling of multiple frames, which reasons the relations between multiple frames and the corresponding cameras to predict the next frame. To facilitate learning and ensure consistency among generated frames, we introduce a locality constraint based on the input cameras to guide self-attention among a large number of patches across space and time. Our method outperforms state-of-the-art view synthesis approaches by a large margin, especially when synthesizing long-term future in indoor 3D scenes. Project page at https://xrenaa.github.io/look-outside-room/.

Scene extrapolation—the idea of generating novel views by flying into a given image—is a promising, yet challenging task. For each predicted frame, a joint inpainting and 3D refinement problem has to be solved, which is ill posed and includes a high level of ambiguity. Moreover, training data for long-range scenes is difficult to obtain and usually lacks sufficient views to infer accurate camera poses. We introduce DiffDreamer, an unsupervised framework capable of synthesizing novel views depicting a long camera trajectory while training solely on internet-collected images of nature scenes. Utilizing the stochastic nature of the guided denoising steps, we train the diffusion models to refine projected RGBD images but condition the denoising steps on multiple past and future frames for inference. We demonstrate that image-conditioned diffusion models can effectively perform long-range scene extrapolation while preserving consistency significantly better than prior GAN-based methods. DiffDreamer is a powerful and efficient solution for scene extrapolation, producing impressive results despite limited supervision. Project page: https://primecai.github.io/diffdreamer.

Despite the progress of 3D Gaussian Splatting (3DGS), reconstructing one-shot animatable 3D human avatars from a single image remains a challenging task. Existing 3DGS-based methods primarily rely on appearance observation and motion cues from monocular videos to reconstruct animatable 3D avatars. However, when applied to single-image setups, these methods struggle to extract accurate 3D features from a 2D image, limiting their ability to capture fine-grained appearance details and dynamic deformation, especially from challenging perspectives. In this work, we propose SIAvatar, a novel single-image human reconstruction method that integrates diffusion-based appearance prior and parametric human model geometry prior within a 3DGS framework. Secondly, a vertex-based adaptive Gaussian densification scheme is introduced to effectively represent human geometry while mitigating artifacts. Extensive experiments demonstrate that SIAvatar generates realistic 3D avatars with plausible appearance details and novel pose animation from a single input image. Project page: https://siavatar.github.io/.

Existing fusion algorithms often struggle to maintain modality features, leading to issues like color distortion and halo artifacts in fusion results. To address these challenges, this paper proposes a multimodal image fusion method based on diffusion models. Specifically, the proposed approach formulates the multimodal fusion task as a conditional generation problem within a DDPM sampling framework. It utilizes multi-scale modality features to guide the iterative generation of fusion images. Our proposed method consists of two key modules: a feature extraction module and a diffusion generation module. The former employs a Swin Transformer network to extract complementary modality features from multimodal images, while the latter implements a denoising network with forward and reverse diffusion processes in pixel space to facilitate data distribution learning. To maintain the original spectral characteristics of objects, enhance modal information integration, and preserve color fidelity, gradient consistency loss and intensity consistency loss are introduced. Extensive experiments validate that our method outperforms some state-of-the-art fusion techniques, especially in terms of color fidelity.

Recent text-to-video diffusion models have achieved impressive progress. In practice, users often desire the ability to control object motion and camera movement independently for customized video creation. However, current methods lack the focus on separately controlling object motion and camera movement in a decoupled manner, which limits the controllability and flexibility of text-to-video models. In this paper, we introduce Direct-a-Video, a system that allows users to independently specify motions for multiple objects as well as camera’s pan and zoom movements, as if directing a video. We propose a simple yet effective strategy for the decoupled control of object motion and camera movement. Object motion is controlled through spatial cross-attention modulation using the model’s inherent priors, requiring no additional optimization. For camera movement, we introduce new temporal cross-attention layers to interpret quantitative camera movement parameters. We further employ an augmentation-based approach to train these layers in a self-supervised manner on a small-scale dataset, eliminating the need for explicit motion annotation. Both components operate independently, allowing individual or combined control, and can generalize to open-domain scenarios. Extensive experiments demonstrate the superiority and effectiveness of our method. Project page and code are available at https://direct-a-video.github.io/.

Image animation has seen significant progress, driven by the powerful generative capabilities of diffusion models. However, maintaining appearance consistency with static input images and mitigating abrupt motion transitions in generated animations remain persistent challenges. While text-to-video (T2V) generation has demonstrated impressive performance with diffusion transformer models, the image animation field still largely relies on U-Net-based diffusion models, which lag behind the latest T2V approaches. Moreover, the quadratic complexity of vanilla self-attention mechanisms in Transformers imposes heavy computational demands, making image animation particularly resource-intensive. To address these issues, we propose MiraMo, a framework designed to enhance efficiency, appearance consistency, and motion smoothness in image animation. Specifically, MiraMo introduces three key elements: (1) A foundational text-to-video architecture replacing vanilla self-attention with efficient linear attention to reduce computational overhead while preserving generation quality; (2) A novel motion residual learning paradigm that focuses on modeling motion dynamics rather than directly predicting frames, improving temporal consistency; and (3) A DCT-based noise refinement strategy during inference to suppress sudden motion artifacts, complemented by a dynamics control module to balance motion smoothness and expressiveness. Extensive experiments against state-of-the-art methods validate the superiority of MiraMo in generating consistent, smooth, and controllable animations with accelerated inference speed. Additionally, we demonstrate the versatility of MiraMo through applications in motion transfer and video editing tasks. The project page is available athttps://maxin-cn.github.io/miramo_project.

3D face generation is a critical task for immersive multimedia applications, where a key challenge is the joint synthesis of expressive geometry and BRDF textures. Existing methods often struggle with geometric-textural coherence and corresponding reflectance modeling. To overcome these limitations, we present ExpDiff, a framework that generates expression meshes and corresponding BRDF textures from a single neutral-expression face. Our method employs an attention-based diffusion model to learn the semantic transition across expressions. To ensure correspondence between geometry and texture, we introduce a unified representation that explicitly models geometric-textural interaction, which is encoded into a shared latent space by models pre-trained on a vast dataset for strong generalization. To achieve semantically coherent and physically consistent generation, we propose to guide the denoising direction with specially designed textual prompts. We further construct two novel facial expression datasets, J-Reflectance, for ultra-high-quality assets, and FFHQ-BRDFExp for diverse identities, both of which are publicly released to advance the community. Extensive experiments demonstrate our method's superior performance in photo-realistic facial expression synthesis. Project page: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://cyh-sj.github.io/expdiff/</uri>.

We introduce the problem of perpetual view generation— long-range generation of novel views corresponding to an arbitrarily long camera trajectory given a single image. This is a challenging problem that goes far beyond the capabilities of current view synthesis methods, which quickly degenerate when presented with large camera motions. Methods for video generation also have limited ability to produce long sequences and are often agnostic to scene geometry. We take a hybrid approach that integrates both geometry and image synthesis in an iterative ‘render, refine and repeat’ framework, allowing for long-range generation that cover large distances after hundreds of frames. Our approach can be trained from a set of monocular video sequences. We propose a dataset of aerial footage of coastal scenes, and compare our method with recent view synthesis and conditional video generation baselines, showing that it can generate plausible scenes for much longer time horizons over large camera trajectories compared to existing methods. Project page at https://infinite-nature.github.io.

We introduce POP3D, a novel framework that creates a full 360° -view 3D model from a single image. POP3D resolves two prominent issues that limit the single-view reconstruction. Firstly, POP3D offers substantial generalizability to arbitrary categories, a trait that previous methods struggle to achieve. Secondly, POP3D further improves reconstruction fidelity and naturalness, a crucial aspect that concurrent works fall short of. Our approach marries the strengths of four primary components: (1) a monocular depth and normal predictor that serves to predict crucial geometric cues, (2) a space carving method capable of demarcating the potentially unseen portions of the target object, (3) a generative model pre-trained on a large-scale image dataset that can complete unseen regions of the target, and (4) a neural implicit surface reconstruction method tailored in reconstructing objects using RGB images along with monocular geometric cues. The combination of these components enables POP3D to readily generalize across various in-the-wild images and generate state-of-the-art reconstructions, outperforming similar works by a significant margin. Project page: http://cg.postech.ac.kr/research/POP3D.

Recently, large pretrained models (e.g., BERT, Style-GAN, CLIP) show great knowledge transfer and generalization capability on various downstream tasks within their domains. Inspired by these efforts, in this paper we propose a unified model for open-domain image editing focusing on color and tone adjustment of open-domain images while keeping their original content and structure. Our model learns a unified editing space that is more semantic, intu-itive, and easy to manipulate than the operation space (e.g., contrast, brightness, color curve) used in many existing photo editing softwares. Our model belongs to the image-to-image translation framework which consists of an image encoder and decoder, and is trained on pairs of before-and-after edited images to produce multimodal outputs. We show that by inverting image pairs into latent codes of the learned editing space, our model can be leveraged for vari-ous downstream editing tasks such as language-guided image editing, personalized editing, editing-style clustering, retrieval, etc. We extensively study the unique properties of the editing space in experiments and demonstrate superior performance on the aforementioned tasks <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code and supplementary material can be found at the project page https://jshi31.github.io/SpaceEdit.

Lift3Dreamer: Boosting Text-Driven Novel View Synthesis via Lifted 3D Inpainting Model from Single Images.

HKUST Electronic Theses Image restoration, editing, and assessment with generative artificial intelligence by Liya Ji thesis 2024 1 online resource (xiii, 92 pages) : illustrations (chiefly color) Generative Artificial Intelligence (AI), such as diffusion models and large language models, can…Read more ›

This research article proposes an integrated AI platform designed to revolutionise the apparel industry. The platform, envisioned as a comprehensive ecosystem, aims to enhance every stage of the apparel value chain, from design ideation to marketing and supply chain management. The architecture is built around three core components: an AI-Powered Design Studio, an Intelligent Production &amp; Supply Chain Backbone, and a Hyper-Personalised Marketing &amp; Engagement Engine. The AI-Powered Design Studio leverages generative AI, deep learning, and computer vision to transform the design process. A Trend Forecasting Engine utilises diverse data sources (social media, e-commerce, runway shows) to predict trends with high accuracy, providing data-driven insights that directly inform the AI Co-Creation Suite. This suite employs GANs, diffusion models, and sketch-to-image translation to generate design variations, refine concepts, and simulate fabric drape and fit, resulting in faster design cycles and commercially viable products. A Hyper-Personalisation Module further enhances this by generating personalised designs tailored to individual customer styles and body measurements, bridging the &amp;quot;aspiration gap&amp;quot; between desired and attainable styles. The Intelligent Production &amp; Supply Chain Backbone focuses on efficiency and sustainability. A Material Optimisation and Waste Reduction System uses computer vision to detect fabric defects and optimise cutting layouts, minimising waste. Predictive inventory management and AI-powered logistics orchestration, combined with blockchain technology for enhanced traceability, create a responsive and transparent supply chain. Automated quality control, utilising computer vision, reduces defects and enables predictive maintenance of machinery, optimising production efficiency. A Sustainability and Circularity Management Dashboard provides a holistic view of environmental and social impact, facilitating data-driven decision-making and transparency for consumers. Finally, the Hyper-Personalised Marketing &amp; Engagement Engine uses AI to deliver tailored experiences. A Personalised Marketing and Dynamic Campaign Engine, powered by a Customer Data Platform (CDP) and a &amp;quot;Latent Style&amp;quot; algorithm, provides personalised product recommendations, marketing messages, and promotions. An Automated Content Generation Engine generates marketing assets (product descriptions, social media posts, email copy) at scale, while a Unified Consumer Insights Platform provides real-time market analysis. Conversational AI, through chatbots and virtual stylists, enhances customer support and creates personalised interactions. The use of 3D digital twins and virtual prototyping, including virtual try-on (VTO) capabilities, enhances consumer engagement and reduces return rates. The article concludes with a phased implementation roadmap, prioritising data infrastructure, key modules with high ROI (such as waste reduction), and subsequent integration of generative design and personalisation. The overall goal is a symbiotic relationship between human creativity and AI&amp;apos;s efficiency, resulting in a future-ready apparel industry characterised by enhanced speed, sustainability, and personalisation. Case studies of industry leaders like Zara, Stitch Fix, H&amp;M, and Nike illustrate the successful application of similar AI strategies.

Most current three-dimensional reconstruction methods often require at least two or more input images to reconstruct the shape of the object. Traditionally, it is hard to reconstruct the 3D shape only from one single input image. In this paper, we propose an effective method to reconstruct the 3D texture surface from a single input image for the texture with similar appearances and reflectance properties. Exactly, the patch matching and optimization methods have been combined to resolve the under-constrained problem of 3D reconstruction from one single image. Experimental results have verified the effectiveness of the proposed method in 3D reconstruction according to realistic perception.