Learning and Generation of Drawing Sequences Using a Deep Network for a Drawing Support System

This paper proposed a learning-based approach to reveal diversity possible appearances under the missing area of an occluded unseen image. In general, there are a lot of possible facial appearances for the missing area; for example, a male with a scarf, it is difficult to predict he has a beard in the covered area or not? In this paper, we propose a novel method for facial image inpainting, which generates the missing facial appearance by conditioning on the observable appearance. Given a trained standard Variational Autoencoder (VAE) for un-occluded face generation. To be specified, we search for the possible set of VAE coding vector for the current occluded input image, and the predicted coding should be robust to the missing area. The possible facial appearance set is then recovered through the decoder of VAE model. Experiments show that our method successfully predicts recovered results in large missing regions; these results are diverse, and all are reasonable to be consistent with the observable facial area, i.e., both the facial geometry and the personal characteristics are preserved.

Introduction: Characterizing the immune contexture is key to understanding the tumor microenvironment, developing biomarkers, and guiding therapeutic strategies. In our previous work on colorectal cancer (CRC), we used a pathologist-guided approach to annotate/count eosinophils and lymphocytes. Their abundance and spatial relationships with tumor cells revealed prognostic significance. While clinically valuable, identifying this demands significant pathologist effort. Advancements in AI, particularly foundational models trained on millions of histology images, have driven significant progress in digital pathology and hold promise to address this gap. However, their application to single-cell tasks remains underexplored. This study investigates the potential of foundation model and compares it with conventional convolutional neural network (CNN) using previously annotated eosinophils and lymphocytes in CRC histology images. Methods: We used 578 CRC cases in The Cancer Genome Atlas with H&E-stained whole-slide images (WSIs) containing 8, 836, 393 cells, including 896, 322 lymphocytes and eosinophils. For evaluation, 109 WSIs were held out as a test set, with 5 WSIs per density quartile selected from our previous study. The foundational model UNI (11.2M parameters) was fine-tuned and a ResNet18 (303M parameters) model was trained. Performance was evaluated using F1-score and AUROC, with Spearman’s correlation accessing WSI-level cell density correlations. Results: For eosinophil, ResNet18 achieved an F1-score of 0.841, AUROC of 0.986, and Spearman's correlation of 0.679, compared to UNI’s 0.847, 0.990, and 0.697, respectively. For lymphocytes, ResNet18 scored 0.697, 0.932, and 0.686, while UNI achieved 0.719, 0.941, and 0.662. ResNet18 performed comparably to UNI despite simpler architecture, at both cell and patient levels. Predicted cell densities generally aligned with the annotated quartiles from previous work, particularly in the 1st and 4th quartiles. Moderate density correlations (0.66-0.69) were observed, but a pathologist's review of misclassified cells revealed imperfections in the semi-supervised cell annotations. Conclusion: The H&E-based AI model for predicting immune cells offers a scalable alternative to manual annotation, enabling automated quantification of existing immune biomarkers like tumor-infiltrating lymphocytes and tumor-associated tissue eosinophilia. Adaptable to other tumor types, it also opens opportunities to discover novel spatial immune biomarkers by leveraging large H&E cohorts and integrating protein-stained immune cell data with histopathological knowledge. Future work will focus on enhancing performance through weakly supervised approaches using carefully selected training cells, and multi-cell type modeling to better capture the morphological characteristics of diverse cell types in a single model. Citation Format: Colwyn Jia Kang Lai, Wei Kit Tan, Marcia Zhang, Timothy Wang, Felicia Wee, Ai Lin Wang, Solomonraj Wilson, Tomotaka Ugai, Joe Sheng Yeong Poh, Jonathan A. Nowak, Shuji Ogino, Juha P. Väyrynen, Mai Chan Lau. Predictive performance comparison of foundational and CNN models for single-cell immune profiling [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 6316.

For multivariate co-generation in scientific applications, we advocate pairwise block rather than joint modeling of all variables. This design mitigates the computational burden and data imbalance. To this end, we propose an Annealed Co-Generation (ACG) framework that replaces high-dimensional diffusion modeling with a low-dimensional diffusion model, which enables multivariate co-generation by composing pairwise variable generations. We first train an unconditional diffusion model over causal variables that are disentangled into pairs. At inference time, we recover the joint distribution by coupling these pairwise models through shared common variables, enabling coherent multivariate generation without any additional training. By employing a three-stage annealing process-Consensus, Heating, and Cooling-our method enforces consistency across shared common variables and progressively constrains each pairwise data distribution to lie on a learnable manifold, while maintaining high likelihood within each pair. We demonstrate the framework's flexibility and efficacy on two distinct scientific tasks: flow-field completion and antibody generation. All datasets and code will be made publicly available upon publication.

Data imputation and data generation have important applications across many domains where incomplete or missing data can hinder accurate analysis and decision-making. Diffusion models have emerged as powerful generative models capable of capturing complex data distributions across various data modalities such as image, audio, and time series. Recently, they have been also adapted to generate tabular data. In this article, we propose a diffusion model for tabular data that introduces three key enhancements: (1) a conditioning attention mechanism, (2) an encoder–decoder transformer as the denoising network, and (3) dynamic masking. The conditioning attention mechanism is designed to improve the model’s ability to capture the relationship between the condition and synthetic data. The transformer layers help model interactions within the condition (encoder) or synthetic data (decoder), while dynamic masking enables our model to efficiently handle both missing data imputation and synthetic data generation tasks within a unified framework. We conduct a comprehensive evaluation by comparing the performance of diffusion models with transformer conditioning against state-of-the-art techniques such as Variational Autoencoders, Generative Adversarial Networks, and Diffusion Models, on benchmark datasets. Our evaluation focuses on the assessment of the generated samples with respect to three important criteria, namely: (1) machine learning efficiency, (2) statistical similarity, and (3) privacy risk mitigation. For the task of data imputation, we consider the efficiency of the generated samples across different levels of missing features. The results demonstrate average superior machine learning efficiency and statistical accuracy compared to the baselines, while maintaining privacy risks at a comparable level, particularly showing increased performance in datasets with a large number of features. By conditioning the data generation on a desired target variable, the model can mitigate systemic biases, generate augmented datasets to address data imbalance issues, and improve data quality for subsequent analysis. This has significant implications for domains such as healthcare and finance, where accurate, unbiased, and privacy-preserving data are critical for informed decision-making and fair model outcomes.

Although deep convolutional neural network based face recognition system has achieved remarkable success, it is susceptible to adversarial images: carefully constructed imperceptible perturbations can easily mislead deep neural networks. A recent study has shown that in addition to regular off-manifold adversarial images, there are also adversarial images on the manifold. In this paper, we propose Adversarial Variational AutoEncoder (A-VAE), a novel framework to tackle both types of attacks. We hypothesize that both off-manifold and on-manifold attacks move the image away from the high probability region of image manifold. We utilize variational autoencoder (VAE) to estimate the lower bound of the log-likelihood of image and explore to project the input images back into the high probability regions of image manifold again. At inference time, our model synthesizes multiple similar realizations of a given image by random sampling, then the nearest neighbor of the given image is selected as the final input of the face recognition model. As a preprocessing operation, our method is attack-agnostic and can adapt to a wide range of resolutions. The experimental results on LFW demonstrate that our method achieves state-of-the-art defense success rate against conventional off-manifold attacks such as FGSM, PGD, and C&W under both grey-box and white-box settings, and even on-manifold attack.

A method combining a variational autoencoder (VAE) and a denoising diffusion probabilistic model (DDPM) is proposed to efficiently compress and reconstruct complex, large-scale atomic coordinates, such as those in polycrystals. Using the atomic coordinates of Ni polycrystalline, we show that the combination of the VAE and DDPM improves reconstruction accuracy compared with the VAE alone. Moreover, the proposed method can reproduce the time evolution of atomic coordinates from MD simulations, even when trained on a single polycrystal at a specific time. In addition, the deviation in the number of atoms from the original coordinates was found to be small across all subregions, including those with various facets and grain boundaries, demonstrating robustness of the model proposed in this study.

Despite the rapid advancements in generative AI, the integration of Large Language Models (LLMs) with diffusion models remains an underexplored domain, with significant potential for transformative applications across multimodal tasks. This survey addresses this critical gap by providing a comprehensive analysis of recent progress in this integration. Methodologically, the study examines approaches like latent space alignment, prompt engineering, and novel architectures that foster synergy between LLMs and diffusion models. Key findings reveal that while this integration enhances generative capabilities, it introduces challenges, including high computational costs, misalignment of model modalities, data scarcity, and quality control issues. The survey systematically evaluates existing solutions to these challenges, highlighting their strengths, limitations, and practical implications. Emerging trends such as efficient fine-tuning strategies, hybrid architectures, and multimodal data augmentation are identified as promising avenues for future research. By synthesizing current knowledge and offering actionable insights, this survey serves as a valuable resource for researchers and practitioners seeking to explore the combined potential of LLMs and diffusion models. The repository for this survey is publicly available at https://github.com/AnasHXH/Connecting-LLMs-to-Diffusion-Models-A-Survey.

This paper reviews the progress in text-to-image generation, which enables the creation of images from textual descriptions. This technology holds promise across various fields, including creative arts, gaming, and healthcare. The main approaches in this area are Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Diffusion Models (DM). While GANs initially made significant advancements in realistic image generation, they faced issues with stability and diversity. VAEs introduced a probabilistic approach, allowing for diverse outputs but often at the cost of image quality. The development of DM, like Stable Diffusion, Imagen, and DALL-E 2, has addressed many limitations, producing high-quality, coherent images through iterative denoising. DM stands out for its stability and ability to generate detailed, semantically accurate images. This review explores the strengths and limitations of each approach, with an emphasis on the advantages of DM. It also discusses future directions, including improving efficiency, enhancing multimodal capabilities, and reducing data requirements to make these models more accessible and versatile for various applications.

Diffusion models have emerged as a popular method for 3D generation. However, it is still challenging for diffusion models to efficiently generate diverse and high‐quality 3D shapes. In this paper, we introduce OctFusion, which can generate 3D shapes with arbitrary resolutions in 2.5 seconds on a single Nvidia 4090 GPU, and the extracted meshes are guaranteed to be continuous and manifold. The key components of OctFusion are the octree‐based latent representation and the accompanying diffusion models. The representation combines the benefits of both implicit neural representations and explicit spatial octrees and is learned with an octree‐based variational autoencoder. The proposed diffusion model is a unified multi‐scale U‐Net that enables weights and computation sharing across different octree levels and avoids the complexity of widely used cascaded diffusion schemes. We verify the effectiveness of OctFusion on the ShapeNet and Objaverse datasets and achieve state‐of‐the‐art performances on shape generation tasks. We demonstrate that OctFusion is extendable and flexible by generating high‐quality color fields for textured mesh generation and high‐quality 3D shapes conditioned on text prompts, sketches, or category labels. Our code and pre‐trained models are available at https://github.com/octree‐nn/octfusion.

By formulating the data generation as a sequence procedure of denoising autoencoding, diffusion models have achieved superior in-painting performance on image data and beyond. Nevertheless, it is not trivial when capitalizing on diffusion models to generate missing 3D points. The difficulty originates from the intrinsic structure where 3D point cloud is a set of unordered and irregular coordinates. That motivates us to delve into the 3D structural information for designing point cloud encoder-decoder and shape latent generator, to precisely formulate the latent distribution of the complete point cloud and partial observation. In this paper, we propose Point cloud completion with Latent Diffusion Models (PointLDM), a new approach that leverages the conditional denoising diffusion probabilistic modeling (DDPM) in the 3D latent space for shape reconstruction. The architecture of PointLDM consists of a transformer-based variational auto-encoder (VAE) to model the complete shape latent, and a diffusion network for shape latent prediction. The encoder of VAE exploits both of global shape latent and local point features in shape distribution learning. With the learnt shape latent, the decoder first decodes the shape latent into coarse points, and then recovers the fine-grained details around each coarse point by deforming a 2D grid. To reconstruct the shape latent from partial observation, the diffusion network treats the partial observation as the conditional input and generates the shape latent via DDPM. Extensive experiments conducted on MVP, Completion3D, and KITTI quantitatively and qualitatively demonstrate the efficacy of PointLDM over the state-of-the-art shape completion approaches.

Dexterous grasp generation that considers armrelated constraints is crucial in real-world scenarios involving armenvironment collision avoidance, workspace boundary grasps, and consecutive grasping. Existing hand-centric grasp models, which primarily focus on the floating hand's pose, are insufficient for such cases. Conventional arm-aware methods either rely on rejection sampling to discard infeasible samples or require retraining on arm-specific data, leading to low sample efficiency under adverse conditions or limited generalization across different robots and environments. To overcome these limitations, this letter presents an arm-aware dexterous grasp generation framework that leverages pretrained arm-agnostic grasp models while integrating arm and environmental information only at inference time. Specifically, we formulate arm-aware constrained grasp generation as a joint optimization of hand pose and arm configuration, and derive closed-form gradients for arm-related constraints. Assuming the hand pose distribution is represented by a diffusion model, we prove that gradient-based optimization is equivalent to guided diffusion sampling, steering near-feasible samples toward the feasible region. Through comprehensive evaluation involving 10k objects across 6 scenarios, we demonstrate that the proposed framework generates feasible grasps in highly constrained settings with significantly higher probability, highlighting its advantages in realworld applications. Supplementary materials and appendix are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://arm-aware-dexgrasp.github.io/</uri>.

Medical image translation with deep learning: Advances, datasets and perspectives.

Distributing datasets containing DDoS attack patterns in software-defined networks (SDNs) is an important task for improving threat detection methods. This paper examines machine learning (MO) approaches for generating synthetic data based on the HLD-DDoSDN dataset. Generative adversarial networks (GAN), variational autoencoders (VAE), resampling methods (SMOTE, ADASYN), and diffusion models are considered. A comparative analysis of methods is carried out in terms of generation quality, computational complexity, and applicability in DDoS attack detection tasks. The results show that hybrid methods combining GAN and SMOTE demonstrate the best balance between data quality and efficiency.

Synthetic artificial intelligence (AI) is rapidly redefining biomedical research-yet in cardiovascular medicine, its clinical relevance remains obscure, underexplored, and underestimated. Unlike traditional AI, which interprets data, synthetic AI generates entirely new, patient-like information: from realistic ECG signals to cardiac imaging and virtual cohorts that simulate disease progression. While recent publications have addressed specific synthetic AI tools in cardiology, no prior review has comprehensively synthesized their architectures, clinical applications, and implementation challenges within a single, practice-oriented framework. This State-of-the-Art Review fills that gap. We provide a clear, critical synthesis of core architectures-Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), Diffusion Models, Transformers, Autoregressive Models, Digital Twins, and Synthetic Cohort Simulators-and map their emerging cardiovascular applications. We examine technical barriers, ethical concerns, regulatory uncertainties, and integration challenges, while anchoring the discussion in real-world clinical priorities. This review is not only a scientific analysis-it is a call to engagement. For academic researchers, it offers conceptual and technical clarity. For clinicians in resource-constrained settings, it presents synthetic AI not as abstract innovation, but as a practical opportunity to enhance diagnostic precision, optimize workflows, and extend clinical insight. Traditional AI supports cardiologists by interpreting data; synthetic AI extends this paradigm by creating new, clinically coherent information that enhances decision-making without replacing physician expertise. As investment grows and methods mature, cardiologists must shape this evolution-not as passive adopters, but as active drivers. This review invites them to take the wheel.

Progress in chemistry has been driven by the streamlining of inverse problem-solving methods. In the history of chemistry, several revolutionary technologies have led to leaps forward: the establishment of atomistic theory in the 19th century, structural analysis by spectroscopy in the 20th century, and the development of simulation by theoretical chemistry. Currently, chemistry is about to make a significant leap forward by integrating generative artificial intelligence (AI). In 2016, deep learning techniques were introduced in this domain, leading to explosive development. This paper reviews the development path, including traditional models such as variational autoencoders and more up-to-date models such as large language models and diffusion models. We also discuss how AI can have a real impact on chemistry, including the possibilities and problems associated with synthesizing AI-generated molecules.