OctFusion: Octree‐based Diffusion Models for 3D Shape Generation

One of the main challenges that the industry faces when dealing with massive data for failure diagnosis is high dimensionality of such data. This can be tackled by dimensionality reduction method such as principal components analysis, which usually results in an improved fault diagnosis. Other available techniques include auto-encoders and its variants denoising auto-encoders and sparse auto-encoders. Most recently, variational auto-encoders are one of the most promising techniques for unsupervised learning with successful applications in image processing and speech recognition. Differently from other auto-encoder methods, variational auto-encoders use variational inference to generate a latent representation of the data and impose a distribution over the latent variables and the data itself. In this article, we propose a fully unsupervised deep variational auto-encoder-based approach for dimensionality reduction in fault diagnosis and explore the variational auto-encoder capabilities for such a task. This is achieved by comparing the latent representations provided by variational auto-encoders to the ones from principal components analysis as well as when no reduction is performed in ball bearings’ fault classification using vibration signals. To tackle massive sensor data, we propose different architectures for the variational auto-encoder’s encoder and decoder that are based on deep neural networks and deep convolutional neural networks. Experiments are also carried out by varying the data preprocessing methods for generating spectrograms and hand-engineering features as well as the use of raw vibration data, the architecture of the neural networks fault classifiers operating on the latent representations from variational auto-encoder and principal components analysis, the degree of data dimension reduction, and the size of the available labeled data used for training the fault classifiers. The results show that variational auto-encoders are a competent and promising tool for dimensionality reduction for use in fault diagnosis and worth further exploring their capabilities beyond vibration signals of ball bearing elements.

Diffusion models have exhibited substantial success in text-to-image generation. However, they often encounter challenges when dealing with complex and dense prompts involving multiple objects, attribute binding, and long descriptions. In this paper, we propose a novel framework called LLM4GEN, which enhances the semantic understanding of text-to-image diffusion models by leveraging the representation of Large Language Models (LLMs). It can be seamlessly incorporated into various diffusion models as a plug-and-play component. A specially designed Cross-Adapter Module (CAM) integrates the original text features of text-to-image models with LLM features, thereby enhancing text-to-image generation. Additionally, to facilitate and correct entity-attribute relationships in text prompts, we develop an entity-guided regularization loss to further improve generation performance. We also introduce DensePrompts, which contains 7,000 dense prompts to provide a comprehensive evaluation for the text-to-image generation task. Experiments indicate that LLM4GEN significantly improves the semantic alignment of SD1.5 and SDXL, demonstrating increases of 9.69% and 12.90% in color on T2I-CompBench, respectively. Moreover, it surpasses existing models in terms of sample quality, image-text alignment, and human evaluation.

In this paper we present a trained diffusion model for image inpainting based on the structural similarity measure. The proposed diffusion model uses several parametrized linear filters and influence functions. Those parameters are learned in a loss based approach, where we first perform a greedy training before conducting a joint training to further improve the inpainting performance. We provide a detailed comparison to state-of-the-art inpainting algorithms based on the TUM-image inpainting database. The experimental results show that the proposed diffusion model is efficient and achieves superior performance. Moreover, we also demonstrate that the proposed method has a texture preserving property, that makes it stand out from previous PDE based methods.

Recent advancements in machine learning comprise generative models such as autoencoders (AE) for learning and compressing 3D data to generate low-dimensional latent representations of 3D shapes. Learning latent representations that disentangle the underlying factors of variations in 3D shapes is an intuitive way to achieve generalization in generative models. However, it remains an open problem to learn a generative model of 3D shapes such that the latent variables are disentangled and represent different interpretable aspects of 3D shapes. In this paper, we propose Split-AE, which is an autoencoder-based architecture for partitioning the latent space into two sets, named as content and style codes. The content code represents global features of 3D shapes to differentiate between semantic categories of shapes, while style code represents distinct visual features to differentiate between shape categories having similar semantic meaning. We present qualitative and quantitative experiments to verify feature disentanglement using our Split-AE. Further, we demonstrate that, given a source shape as an initial shape and a target shape as a style reference, the trained Split-AE combines the content of a source and style of a target shape to generate a novel augmented shape, that possesses the distinct features of the target shape category yet maintains the similarity of the global features with the source shape. We conduct a qualitative study showing that the augmented shapes exhibit a realistic interpretable mixture of content and style features across different shape classes with similar semantic meaning.

A variety of image generation methods have emerged in recent years, notably DALL-E 2, Imagen and Stable Diffusion. While they have been shown to be capable of producing photorealistic images from text prompts facilitated by generative diffusion models conditioned on language input, their capacity for materials design has not yet been explored. Here, we use a trained Stable Diffusion model and consider it as an experimental system, examining its capacity to generate novel material designs especially in the context of 3D material architectures. We demonstrate that this approach offers a paradigm to generate diverse material patterns and designs, using human-readable language as input, allowing us to explore a vast nature-inspired design portfolio for both novel architectured materials and granular media. We present a series of methods to translate 2D representations into 3D data, including movements through noise spaces via mixtures of text prompts, and image conditioning. We create physical samples using additive manufacturing and assess material properties of materials designed via a coarse-grained particle simulation approach. We present case studies using images as starting point for material generation; exemplified in two applications. First, a design for which we use Haeckel’s classic lithographic print of a diatom, which we amalgamate with a spider web. Second, a design that is based on the image of a flame, amalgamating it with a hybrid of a spider web and wood structures. These design approaches result in complex materials forming solids or granular liquid-like media that can ultimately be tuned to meet target demands.

Posterior collapse is a pervasive issue in Variational Autoencoders (VAEs) that leads to the learned latent representations becoming trivial and devoid of meaningful information. To address this problem, this paper presents a novel β-VAE approach, which incorporates a hyperparameter β to strike an optimal balance between the reconstruction loss and the KL divergence loss. By conducting a comprehensive series of experiments and drawing comparisons with existing methods, robust evidence is provided that the proposed β-VAE method effectively mitigates posterior collapse and yields more expressive and informative latent representations.The experimental setup involves various architectures and datasets to demonstrate the versatility and efficacy of the β-VAE approach in diverse settings. Additionally, ablation studies are performed to investigate the impact of different β values on the model's performance, elucidating the role of this hyperparameter in controlling the trade-off between reconstruction quality and latent representation expressiveness. Furthermore, the disentanglement properties of the learned latent space are analyzed, which is a crucial aspect of VAEs, especially when applied to complex, real-world data.In-depth analysis of the results offers valuable insights into the underlying mechanisms of β-VAE, thereby contributing to a more profound understanding of VAEs and their inherent limitations. The findings not only establish the effectiveness of the β-VAE method in preventing posterior collapse but also pave the way for future research on improving VAEs' performance in various applications. Potential future work could explore alternative techniques for balancing the competing objectives of reconstruction and latent representation learning or delve into the theoretical properties of β-VAE, providing a more rigorous foundation for this approach.

Resting state Functional Magnetic Resonance Imaging fMRI (rs-fMRI) has been used extensively to study brain function in psychiatric disorders, yielding insights into brain organization. However, the high dimensionality of the rs-fMRI data presents significant challenges for data analysis. Variational autoencoders (VAEs), a type of neural network, have been instrumental in extracting low-dimensional latent representations of resting state functional connectivity (rsFC) patterns, thereby addressing the complex nonlinear structure of rs-fMRI data. Despite these advances, interpreting these latent representations remains a challenge. This paper aims to address this gap by developing explainable VAE models and testing their utility using rs-fMRI data in autism spectrum disorder (ASD). One-thousand one hundred and fifty participants (601 healthy controls [HC] and 549 patients with ASD) were included in the analysis. RsFC correlation matrices were extracted from the preprocessed rs-fMRI data using the Power atlas, which includes 264 regions of interest (ROIs). Then VAEs were trained in an unsupervised manner. Lastly, we introduce our latent contribution scores to explain the relationship between estimated representations and the original rs-fMRI brain measures. We quantified the latent contribution scores for both the ASD and HC groups at the network level. We found that both ASD and HC groups share the top network connectivitives contributing to all estimated latent components. For example, latent 0 was driven by rsFC within ventral attention network (VAN) in both the ASD and HC. However, we found significant differences in the latent contribution scores between the ASD and HC groups within the VAN for latent 0 and the sensory/somatomotor network for latent 2. This study introduced latent contribution scores to interpret nonlinear patterns identified by VAEs. These scores effectively capture changes in each observed rsFC feature as the estimated latent representation changes, enabling an explainable deep learning model that better understands the underlying neural mechanisms of ASD.

The task of 3D shape generation for realistic data is an important challenge that needs to be addressed in the domain of computer vision, robotics, and graphics which serve as a building block for many real-time applications like autonomous driving or 3D modeling, etc. Estimating shapes from a few 2D images are fundamentally ill-posed as numerous 3D shapes can be explained by a few images. In the absence of complete information, recently, deep learning has been used to fill in the gap by leveraging data driven category level priors. In this work, we propose a novel 3D shape estimation network that uses an image transformer to better encode the shape features into a latent representation which is later decoded using a multilayer perceptron. Our experiments show that image transformers are better than convolution-based encoders due to their wide attention capability. We perform both qualitative and quantitative experiments to demonstrate the effect of new architecture on shape quality and detail.KeywordsComputer visionMultiview shape3D reconstructionShape estimationMultiview imagesImage transformers

In the real world, forecasts would always be based on historical data with the assumption that the behaviour be the same for the future. But how do we forecast when there is no such data available? New product or new technologies normally has limited amount of data available. Knowing that forecasting is valuable for decision making, this paper presents forecasting of new product or new technologies using aggregate diffusion models and modified Bass Model. A newly launched Proton car and its penetration was chosen to demonstrate the possibility of forecasting sales demand where there is limited or no data available. The model was developed to forecast diffusion of new vehicle or an innovation in the Malaysian society. It is to represent the level of spread on the new vehicle among a given set of the society in terms of a simple mathematical function that elapsed since the introduction of the new product. This model will forecast the car sales volume. A procedure of the proposed diffusion model was designed and the parameters were estimated. Results obtained by applying the proposed diffusion model and numerical calculation shows that the model is robust and effective for forecasting demand of the new vehicle. The results reveal that newly developed modified Bass diffusion of demand function has significantly contributed for forecasting the diffusion of new Proton car or new product.

Traditional diffusivity based denoising models detect edges by the gradients of intensities, and thus are easily affected by noise. In this paper, we develop a nonlinear diffusion denoising method which adapts to the local context and thus preserves edges and diffuses more in the smooth regions. In the proposed diffusion model, the modulus of gradient in a diffusivity function is substituted by the modulus of a wavelet detail coefficient and the diffusion of wavelet coefficients is performed based on the local context. The local context is derived directly by analyzing the energy of transform across the scales and thus it performs efficiently in the real-time. The redundant representation of the stationary wavelet transform (SWT) and its shift-invariance lend themselves to edge detection and denoising applications. The proposed stationary wavelet context-based diffusivity (SWCD) model produces a better quality image compared to that attained by two high performance diffusion models, i.e. higher Peak Signal-to-Noise Ratio on average and lesser artifacts and blur are observed in a number of images representing texture, strong edges and smooth backgrounds.

Background Pulmonary hypertension (PH) in newborns is a severe condition characterized by elevated pulmonary artery pressure, leading to right ventricular strain and impaired oxygenation, and significant morbidity and mortality. Right heart catheterization (RHC) is the gold standard for PH diagnosis but is invasive, costly, and associated with procedural risks. Instead, transthoracic echocardiography (ECHO) is widely used for non-invasive PH assessment, although its interpretation is highly operator-dependent. While automated machine learning methods have been proposed for PH detection, existing approaches largely focus on adults and often rely on single-view echocardiographic frames, limiting their generalizability to neonatal populations. Objective We aim to leverage a multi-view variational autoencoder (VAE) framework to predict PH severity in newborns using echocardiographic videos. We hypothesize that learning latent representations across multiple echocardiographic views improves classification performance and generalization compared to traditional models. Methods We analyzed a dataset of 936 transthoracic ECHO videos from 192 newborns (56±160 days old, 2.9±1.5Kg), with a held-out test set of 375 videos from 78 newborns (34±42 days old, 2.1±1.2Kg). Standard ECHO views included PLAX, A4C, and three parasternal short-axis views (PSAX-P, PSAX-S, PSAX-A). PH severity was categorized as none, mild, or moderate-to-severe based on septal flattening patterns. A multi-view VAE framework with a variational mixture-of-experts prior was trained to extract latent representations from ECHO videos, leveraging shared information across different views. Classification performance was compared against single-view and supervised learning from Ragnarsdottir et al. 24’ [1]. The study and method pipeline is illustrated in figure 1. Results Multi-view learning significantly improved PH classification accuracy compared to single-view approaches. Our VAE model achieved an AUROC of 0.83 ± 0.02 and a balanced accuracy of 0.74 ± 0.02 for binary PH detection, outperforming single-view VAEs and traditional supervised learning models. Importantly, our VAE exhibited greater robustness to unseen test data, demonstrating improved generalization across newborn populations. The model effectively distinguished PH severity levels, with the highest performance gains observed in the multi-class classification task (AUROC = 0.74 ± 0.01) –comprehensive results in figure 2. Conclusion Automated PH assessment using echocardiography is essential for early detection and intervention, particularly in neonates where timely diagnosis can impact survival. Our multi-view VAE-based framework enhances PH classification, reduces operator dependency, and improves diagnostic consistency. These improvements could enable more standardized and accessible screening, particularly in resource-limited settings where expert evaluation may not always be available.Study pipeline and method. Summary results (baseline [1]).

BackgroundEnormous clinical and biomedical researches have demonstrated that microbes are crucial to human health. Identifying associations between microbes and diseases can not only reveal potential disease mechanisms, but also facilitate early diagnosis and promote precision medicine. Due to the data perturbation and unsatisfactory latent representation, there is a significant room for improvement.ResultsIn this work, we proposed a novel framework, Multi-scale Variational Graph AutoEncoder embedding Wasserstein distance (MVGAEW) to predict disease-related microbes, which had the ability to resist data perturbation and effectively generate latent representations for both microbes and diseases from the perspective of distribution. First, we calculated multiple similarities and integrated them through similarity network confusion. Subsequently, we obtained node latent representations by improved variational graph autoencoder. Ultimately, XGBoost classifier was employed to predict potential disease-related microbes. We also introduced multi-order node embedding reconstruction to enhance the representation capacity. We also performed ablation studies to evaluate the contribution of each section of our model. Moreover, we conducted experiments on common drugs and case studies, including Alzheimer’s disease, Crohn’s disease, and colorectal neoplasms, to validate the effectiveness of our framework.ConclusionsSignificantly, our model exceeded other currently state-of-the-art methods, exhibiting a great improvement on the HMDAD database.

By formulating the data generation as a sequence procedure of denoising autoencoding, diffusion models have achieved superior in-painting performance on image data and beyond. Nevertheless, it is not trivial when capitalizing on diffusion models to generate missing 3D points. The difficulty originates from the intrinsic structure where 3D point cloud is a set of unordered and irregular coordinates. That motivates us to delve into the 3D structural information for designing point cloud encoder-decoder and shape latent generator, to precisely formulate the latent distribution of the complete point cloud and partial observation. In this paper, we propose Point cloud completion with Latent Diffusion Models (PointLDM), a new approach that leverages the conditional denoising diffusion probabilistic modeling (DDPM) in the 3D latent space for shape reconstruction. The architecture of PointLDM consists of a transformer-based variational auto-encoder (VAE) to model the complete shape latent, and a diffusion network for shape latent prediction. The encoder of VAE exploits both of global shape latent and local point features in shape distribution learning. With the learnt shape latent, the decoder first decodes the shape latent into coarse points, and then recovers the fine-grained details around each coarse point by deforming a 2D grid. To reconstruct the shape latent from partial observation, the diffusion network treats the partial observation as the conditional input and generates the shape latent via DDPM. Extensive experiments conducted on MVP, Completion3D, and KITTI quantitatively and qualitatively demonstrate the efficacy of PointLDM over the state-of-the-art shape completion approaches.

Data imputation and data generation have important applications across many domains where incomplete or missing data can hinder accurate analysis and decision-making. Diffusion models have emerged as powerful generative models capable of capturing complex data distributions across various data modalities such as image, audio, and time series. Recently, they have been also adapted to generate tabular data. In this article, we propose a diffusion model for tabular data that introduces three key enhancements: (1) a conditioning attention mechanism, (2) an encoder–decoder transformer as the denoising network, and (3) dynamic masking. The conditioning attention mechanism is designed to improve the model’s ability to capture the relationship between the condition and synthetic data. The transformer layers help model interactions within the condition (encoder) or synthetic data (decoder), while dynamic masking enables our model to efficiently handle both missing data imputation and synthetic data generation tasks within a unified framework. We conduct a comprehensive evaluation by comparing the performance of diffusion models with transformer conditioning against state-of-the-art techniques such as Variational Autoencoders, Generative Adversarial Networks, and Diffusion Models, on benchmark datasets. Our evaluation focuses on the assessment of the generated samples with respect to three important criteria, namely: (1) machine learning efficiency, (2) statistical similarity, and (3) privacy risk mitigation. For the task of data imputation, we consider the efficiency of the generated samples across different levels of missing features. The results demonstrate average superior machine learning efficiency and statistical accuracy compared to the baselines, while maintaining privacy risks at a comparable level, particularly showing increased performance in datasets with a large number of features. By conditioning the data generation on a desired target variable, the model can mitigate systemic biases, generate augmented datasets to address data imbalance issues, and improve data quality for subsequent analysis. This has significant implications for domains such as healthcare and finance, where accurate, unbiased, and privacy-preserving data are critical for informed decision-making and fair model outcomes.

Purpose:There has been burgeoning interest in applying machine learning methods for predicting radiotherapy outcomes. However, the imbalanced ratio of a large number of variables to a limited sample size in radiation oncology constitutes a major challenge. Therefore, dimensionality reduction methods can be a key to success. The study investigates and contrasts the application of traditional machine learning methods and deep learning approaches for outcome modeling in radiotherapy. In particular, new joint architectures based on variational autoencoder (VAE) for dimensionality reduction are presented and their application is demonstrated for the prediction of lung radiation pneumonitis (RP) from a large-scale heterogeneous dataset.Methods:A large-scale heterogeneous dataset containing a pool of 230 variables including clinical factors (e.g., dose, KPS, stage) and biomarkers (e.g., single nucleotide polymorphisms (SNPs), cytokines and micro-RNAs) in a population of 106 non-small cell lung cancer (NSCLC) patients who received radiotherapy was used for modeling RP. 22 patients had grade 2 or higher RP. Four methods were investigated, including feature selection (case A) and feature extraction (case B) with traditional machine learning methods, a VAE-MLP joint architecture (case C) with deep learning and lastly, the combination of feature selection and joint-architecture (case D). For feature selection, Random forest (RF), Support Vector Machine (SVM) and multi-layer perceptron (MLP) were implemented to select relevant features. Specifically, each method was run for multiple times to rank features within several cross-validated (CV) resampled sets. A collection of ranking lists were then aggregated by top 5% and Kemeny graph methods to identify the final ranking for prediction. A synthetic minority oversampling technique (SMOTE) was applied to correct for class imbalance during this process. For deep learning, a VAE-MLP joint architecture where a VAE aimed for dimensionality reduction and an MLP aimed for classification was developed. In this architecture, reconstruction loss and prediction loss were combined into a single loss function to realize simultaneous training and weights were assigned to different classes to mitigate class imbalance. To evaluate the prediction performance and conduct comparisons, the area under receiver operating characteristic (ROC) curves (AUCs) were performed for nested CVs for both handcrafted feature selections and the deep learning approach. The significance of differences in AUCs was assessed using the DeLong test of U-statistics.Results:An MLP-based method using weight pruning (WP) feature selection yielded the best performance among the different hand-crafted feature selection methods (case A), reaching an AUC of 0.804 (95% CI: 0.761–0.823) with 29 top features. A VAE-MLP joint architecture (case C) achieved a comparable but slightly lower AUC of 0.781 (95% CI: 0.737–0.808) with the size of latent dimension being 2. The combination of handcrafted features (case A) and latent representation (case D) achieved a significant AUC improvement of 0.831 (95% CI: 0.805–0.863) with 22 features (p-value = 0.000642 compared with handcrafted features only (Case A) and p-value=0.000453 compared to VAE alone (Case C)) with an MLP classifier.Conclusion:The potential for combination of traditional machine learning methods and deep learning VAE techniques has been demonstrated for dealing with limited datasets in modeling radiotherapy toxicities. Specifically, latent variables from a VAE-MLP joint architecture are able to complement handcrafted features for the prediction of RP and improve prediction over either method alone.