Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference

This study focuses on harnessing deep-learning-based text-to-image transformation techniques to help webtoon creators’ creative outputs. We converted publicly available datasets (e.g., MSCOCO) into a multimodal webtoon dataset using CartoonGAN. First, the dataset was leveraged for training contrastive language image pre-training (CLIP), a model composed of multi-lingual BERT and a Vision Transformer that learnt to associate text with images. Second, a pre-trained diffusion model was employed to generate webtoons through text and text-similar image input. The webtoon dataset comprised treatments (i.e., textual descriptions) paired with their corresponding webtoon illustrations. CLIP (operating through contrastive learning) extracted features from different data modalities and aligned similar data more closely within the same feature space while pushing dissimilar data apart. This model learnt the relationships between various modalities in multimodal data. To generate webtoons using the diffusion model, the process involved providing the CLIP features of the desired webtoon’s text with those of the most text-similar image to a pre-trained diffusion model. Experiments were conducted using both single- and continuous-text inputs to generate webtoons. In the experiments, both single-text and continuous-text inputs were used to generate webtoons, and the results showed an inception score of 7.14 when using continuous-text inputs. The text-to-image technology developed here could streamline the webtoon creation process for artists by enabling the efficient generation of webtoons based on the provided text. However, the current inability to generate webtoons from multiple sentences or images while maintaining a consistent artistic style was noted. Therefore, further research is imperative to develop a text-to-image model capable of handling multi-sentence and -lingual input while ensuring coherence in the artistic style across the generated webtoon images.

Synthesis of a realistic image from matching visual descriptions provided in the textual format is a challenge that has attracted attention in the recent research community in the field of artificial intelligence. Generation of the image from given text input is a problem, where given a text input, an image which matches text description must be generated. However, a relatively new class of convolutional neural networks referred to as generative adversarial networks (GANs) has provided compelling results in understanding textual features and generating high-resolution images. In this work, the main aim is to generate an automobile image from the given text input using generative adversarial networks and manipulate automobile colour using text-adaptive discriminator. This work involves creating a detailed text description of each image of a car to train the GAN model to produce images. KeywordsGenerative adversarial networksGeneratorDiscriminatorGANSText to image synthesis

Diffusion-based text-to-image generation models like GLIDE and DALLE-2 have gained wide success recently for their superior performance in turning complex text inputs into images of high quality and wide diversity. In particular, they are proven to be very powerful in creating graphic arts of various formats and styles. Although current models supported specifying style formats like oil painting or pencil drawing, fine-grained style features like color distributions and brush strokes are hard to specify as they are randomly picked from a conditional distribution based on the given text input. Here we propose a novel style guidance method to support generating images using arbitrary style guided by a reference image. The generation method does not require a separate style transfer model to generate desired styles while maintaining image quality in generated content as controlled by the text input. Additionally, the guidance method can be applied without a style reference, denoted as self style guidance, to generate images of more diverse styles. Comprehensive experiments prove that the proposed method remains robust and effective in a wide range of conditions, including diverse graphic art forms, image content types and diffusion models.

This paper reviews the progress in text-to-image generation, which enables the creation of images from textual descriptions. This technology holds promise across various fields, including creative arts, gaming, and healthcare. The main approaches in this area are Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Diffusion Models (DM). While GANs initially made significant advancements in realistic image generation, they faced issues with stability and diversity. VAEs introduced a probabilistic approach, allowing for diverse outputs but often at the cost of image quality. The development of DM, like Stable Diffusion, Imagen, and DALL-E 2, has addressed many limitations, producing high-quality, coherent images through iterative denoising. DM stands out for its stability and ability to generate detailed, semantically accurate images. This review explores the strengths and limitations of each approach, with an emphasis on the advantages of DM. It also discusses future directions, including improving efficiency, enhancing multimodal capabilities, and reducing data requirements to make these models more accessible and versatile for various applications.

GMF-GAN: Gradual multi-granularity semantic fusion GAN for text-to-image synthesis

In this paper we tackle the problem of image search when the query is a short textual description of the image the user is looking for. We choose to implement the actual search process as a similarity search in a visual feature space, by learning to translate a textual query into a visual representation. Searching in the visual feature space has the advantage that any update to the translation model does not require to reprocess the, typically huge, image collection on which the search is performed. We propose Text2Vis, a neural network that generates a visual representation, in the visual feature space of the fc6-fc7 layers of ImageNet, from a short descriptive text. Text2Vis optimizes two loss functions, using a stochastic loss-selection method. A visual-focused loss is aimed at learning the actual text-to-visual feature mapping, while a text-focused loss is aimed at modeling the higher-level semantic concepts expressed in language and countering the overfit on non-relevant visual components of the visual loss. We report preliminary results on the MS-COCO dataset.

Target detection in wide-band SAR image faces problems with speckle noise and small target detection. This paper uses the DiffusionDet model to solve the problem. DiffusionDet transforms target detection into a denoising diffusion process from noise box to target box, which consists of an image encoder and a detection decoder. When implemented, the pre-trained image encoder is first used to extract SAR image features. Then, the diffusion model is applied to generate candidate positions for target detection boxes on the feature map. By adding noise to the true value box and learning the inversion process, the target box is gradually refined. In the training phase, a SAR image dataset with true value annotations is used for training. Finally, the optimized DiffusionDet model is applied to the wide-band SAR image target detection task, and the detection performance is improved by adjusting parameters. The maximum IoU value of the DiffusionDet model reaches 1, and the average detection time is only 289.325 milliseconds, which is far lower than Faster R-CNN's <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{5 3 4. 1 2 5}$</tex> milliseconds. In addition, the ablation experiment also reveals the trade-off between the number of sampling times and the number of iterations on detection accuracy and inference speed. The DiffusionDet model effectively solves the problem of speckle noise and small target detection in wide-band SAR image target detection, and provides a new method for SAR image target detection.

Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) are two essential neuroimaging modalities that offer complementary anatomical and functional information for diagnosing brain diseases. However, PET imaging and some advance MRI are expensive and exposes patients to ionizing radiation, limiting its routine clinical use. This thesis explores a novel diffusion model-based framework for synthetic these advanced brain image modalities that aims to reduce healthcare costs and minimize patient pain by replacing the need for real scans with high-fidelity synthetic alternatives. The first contribution of this work addresses the limitations of existing deep learning approaches that typically require separate models for each PET tracer. To overcome this inflexibility, we propose a unified diffusion framework capable of generating multiple types of PET images from multi-sequence MRI inputs within a single training. The model incorporates a cross-attention-based encoder to capture rich inter-modality dependencies and leverages a categorical embedding to condition the generation on specific tracer types, enabling flexible and efficient multi-tracer PET synthesis. The second contribution tackles the frequent issue of missing MRI modalities in clinical practice. We introduce FMM-Diff (Feature Mapping and Merging Diffusion Model), a modality-robust generative architecture designed to synthesize accurate PET images even when some MRI sequences are unavailable. Instead of relying on shared encoders or basic masking strategies, FMM-Diff employs modality-specific encoders and feature inference modules to reconstruct missing modality representations. A subsequent merging module adaptively integrates cross-modal information, ensuring stable and accurate image generation under incomplete input conditions. Together, these contributions advance the field of synthetic neuroimaging by enhancing the flexibility, scalability, and robustness of PET generation from MRI using diffusion models, paving the way for safer, more accessible, and cost-effective diagnostic solutions.

Rain removal from a single image is a challenge which has been studied for a long time. In this paper, a novel convolutional neural network based on wavelet and dark channel is proposed. On one hand, we think that rain streaks correspond to high frequency component of the image. Therefore, haar wavelet transform is a good choice to separate the rain streaks and background to some extent. More specifically, the LL subband of a rain image is more inclined to express the background information, while HL, LH subband tend to represent the rain streaks and the edges respectively. On the other hand, the accumulation of rain streaks from long distance makes the rain image look like haze veil. We extract dark channel of rain image as a feature map in network. By increasing this mapping between the dark channel of input and output images, we achieve haze removal in an indirect way. All of the parameters are optimized by back-propagation. Experiments on both synthetic and realworld datasets reveal that our method outperforms other state-of-the-art methods from a qualitative and quantitative perspective.

Change detection methods aim to identify significantly changed areas in co-registered bitemporal images taken of the same area. Since not only do bitemporal images usually have different environmental conditions ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> different weather conditions, noises, and seasonal changes) but also changes irrelevant to the purpose of change detection ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.,</i> road changes when detecting building change), which should not be detected as changed areas, change detection methods often suffer from the problem of pseudo-change detection. To alleviate this problem, we propose an encoder-decoder-based Siamese network with a channel-wise co-attention module that considers the channel-wise correlations between a feature map in one image and all feature maps in the other image. By comparing the feature map in one image with the revised feature map in the other image considering the correlations, we are able to reduce the differences between the feature maps when pseudo-changes exist, thereby rendering the proposed method more robust to pseudo-changes. In addition, we apply a contrastive loss function that encourages the pairs of feature maps corresponding to unchanged regions to be similar, which can help improve the performance of change detection. We verified the performance of the proposed method through experiments using datasets such as the change detection dataset (CDD) and building change detection dataset (BCDD). In the experiment, the proposed method achieved significantly improved performance compared with existing methods in terms of recall, precision, f1-score, and overall accuracy.

Generative AI models such as Generative Adversarial Networks (GANs) and Diffusion Models (DMs), have demonstrated remarkable capabilities in producing high-quality synthetic data for applications ranging from image synthesis and medical imaging to data augmentation. However, the complex model architectures and unique computational operations pose significant challenges for traditional electronic accelerators. To address energy/sustainability bottlenecks with conventional electronic hardware, we present a novel silicon photonic accelerator targeting both GANs and DMs. Experimental evaluations show that our photonic accelerator achieves at least 2.18× lower energy consumption and at least 4.4× throughout improvement compared to several state-of-theart CPU, GPU, FPGA, ReRAM, and ASIC-based accelerators.

People can describe spatial scenes with language and, vice versa, create images based on linguistic descriptions. However, current systems do not even come close to matching the complexity of humans when it comes to reconstructing a scene from a given text. Even the ever-advancing development of better and better Transformer-based models has not been able to achieve this so far. This task, the automatic generation of a 3D scene based on an input text, is called text-to-3D scene generation. The key challenge, and focus of this dissertation, now relate to the following topics: (a) Analyses of how well current language models understand spatial information, how static embeddings compare, and whether they can be improved by anaphora resolution. (b) Automated resource generation for context expansion and grounding that can help in the creation of realistic scenes. (c) Creation of a VR-based text-to-3D scene system that can be used as an annotation and active-learning environment, but can also be easily extended in a modular way with additional features to solve more contexts in the future. (d) Analyze existing practices and tools for digital and virtual teaching, learning, and collaboration, as well as the conditions and strategies in the context of VR. In the first part of this work, we could show that static word embeddings do not benefit significantly from pronoun substitution. We explain this result by the loss of contextual information, the reduction in the relative occurrence of rare words, and the absence of pronouns to be substituted. But we were able to we have shown that both static and contextualizing language models appear to encode object knowledge, but require a sophisticated apparatus to retrieve it. The models themselves in combination with the measures differ greatly in terms of the amount of knowledge they allow to extract. Classifier-based variants perform significantly better than the unsupervised methods from bias research, but this is also due to overfitting. The resources generated for this evaluation are later also an important component of point three. In the second part, we present AffordanceUPT, a modularization of UPT trained on the HICO-DET dataset, which we have extended with Gibsonien/telic annotations. We then show that AffordanceUPT can effectively make the Gibsonian/telic distinction and that the model learns other correlations in the data to make such distinctions (e.g., the presence of hands in the image) that have important implications for grounding images to language. The third part first presents a VR project to support spatial annotation respectively IsoSpace. The direct spatial visualization and the immediate interaction with the 3D objects should make the labeling more intuitive and thus easier. The project will later be incorporated as part of the Semantic Scene Builder (SeSB). The project itself in turn relies on the Text2SceneVR presented here for generating spatial hypertext, which in turn is based on the VAnnotatoR. Finally, we introduce Semantic Scene Builder (SeSB), a VR-based text-to-3D scene framework using Semantic Annotation Framework (SemAF) as a scheme for annotating semantic relations. It integrates a wide range of tools and resources by utilizing SemAF and UIMA as a unified data structure to generate 3D scenes from textual descriptions and also supports annotations. When evaluating SeSB against another state-of-the-art tool, it was found that our approach not only performed better, but also allowed us to model a wider variety of scenes. The final part reviews existing practices and tools for digital and virtual teaching, learning, and collaboration, as well as the conditions and strategies needed to make the most of technological opportunities in the future.

Semantic picture synthesis from natural language descriptions is a difficult but important task at the junction of CV and NLP. Using Adaptive Multimodal Generative Adversarial Networks (GANs), this work proposes a new approach for creating high-fidelity and semantically coherent images from textual descriptions. Using a dynamic attention mechanism (DAM), the model aggregates textual and visual modalities, enabling the generator to focus on relevant linguistic traits and dynamically produce images. Moreover, a multi-stage refinement (MSR) process ensures that image features progressively align with the input text. In addition to visual realism, the approach offers a modality-aware discriminator assessing semantic conformity with the description. Extensive tests on benchmark datasets reveal that the method outperforms existing models in terms of variety, text-image consistency (TIC), and picture quality. This work opens exciting future directions, including design prototyping, art generation, and content creation assistance.

Progress in Text-to-Image (T2I) models has significantly advanced the generation of images from textual descriptions. Existing metrics, such as CLIP, effectively measure the semantic alignment between single prompts and their corresponding images. However, they fall short in evaluating a model's ability to generalize across a broad spectrum of textual inputs. To address this gap, we propose the VLEU (Visual Language Evaluation Understudy) metric. VLEU leverages the power of Large Language Models (LLMs) to sample from the visual text domain, encompassing the entire range of potential inputs for the T2I task, to generate a wide variety of visual text. The images generated by T2I models from these prompts are then assessed for their alignment with the input text using the CLIP model. VLEU quantitatively measures a model's generalizability by computing the Kullback-Leibler (KL) divergence between the visual text marginal distribution and the conditional distribution over the images generated by the model. This provides a comprehensive metric for comparing the overall generalizability of T2I models, beyond single-prompt evaluations, and offers valuable insights during the finetuning process. Our experimental results demonstrate VLEU's effectiveness in evaluating the generalizability of various T2I models, positioning it as an essential metric for future research and development in image synthesis from text prompts. Our code and data will be publicly available at https://github.com/mio7690/VLEU.

The long-term durability of concrete is essential for sustainable construction and reducing carbon emissions. A key factor affecting durability is the ingress of hazardous substances, making the understanding of ionic diffusion in concrete crucial. While ionic diffusion is significantly affected by the microstructure of cement paste, existing diffusion models often oversimplify this microstructure by neglecting particle internal pores (PIPs). This simplification can lead to inaccuracies in predicting ionic diffusivity and, consequently, the long-term performance of concrete structures. This study addresses this critical gap by developing a hydration-diffusion simulation method to investigate the effects of PIPs on ionic diffusion in hydrated tricalcium silicate (C3S), a primary constituent of cement clinker. Our simulations revealed that: (1) the effects of PIP size distribution and particle porosity (PP) on ionic diffusion depended on the water content, water-to-solid ratio, solid C3S content, and hydration age; (2) at early ages, in hydration systems with a fixed C3S content or water-to-solid ratio, higher PP would result in lower ionic diffusivity in hydrated C3S paste. However, contrasting conclusions emerged under conditions of fixed water content; (3) the effects of PIPs on the ionic diffusion in hydrated C3S also varied with hydration age and water-to-solid ratio.