A Study on Webtoon Generation Using CLIP and Diffusion Models

Entity matching (EM) aims to identify equivalent entities across different data sources. Current EM assumes that these data are either homogeneous with aligned schema or heterogeneous but can be transformed into a unified modality. There is an urgent need to consider the entities with different modalities to support practical application scenarios over data lakes such as multi-modal data integration and recommendation system. It is impractical to unify their data modalities. To support EM on heterogeneous entity with different data formats and modalities, we propose cross-modal entity matching in this paper. Inspired by the promising performance achieved by recent pre-trained models, we perform cross-modal entity matching by prompt-tuning pre-trained multi-modal large models (MMLMs) in an unsupervised manner. However, the prompt-tuning faces three challenging issues: (i) objective gap between pre-training and tuning of MMLMs; (ii) data modality gap between the inputs of MMLMs and our matching task; (iii) prompt efficiency on large data. Therefore, we firstly propose a novel EM framework (namely, CrossEM) that addresses cross-modal EM as a matching probability problem with specific prompt-tuning. Secondly, two alternative prompt generation methods are designed to extract structural knowledge from heterogeneous data to overcome the data modality gap with pre-trained models. Thirdly, we present an improved matching framework (namely, CrossEM+) to boost the prompt efficiency on large heterogeneous data. Experimental evaluations verify that our methods significantly outperform the state-of-the-art approaches on three benchmarks. Furthermore, our case study highlights the considerable potential of cross-modal EM in improving the performance of downstream tasks, thereby benefitting a wider range of research areas.

Understanding the world in three dimensions has long been a scientific challenge with significant practical implications. In the era of deep learning, researchers have made substantial progress in developing 3D representations using deep neural networks. Among the myriad of entities in our complex environment, humans stand out due to their general significance in science and their specific relevance in various applications. This thesis focuses on human-centric 3D representation learning within the framework of deep learning. Specifically, we explore three key areas: human-centric 3D perception, human reconstruction, and 3D human generation. The thesis presents five studies that collectively address these topics. In the realm of human-centric 3D perception, we introduce a versatile multi-modal pre-training approach. By harnessing the diverse modalities of human data, such as RGB images, depth, and 2D keypoints, we present a general framework HCMoCo for effective human-centric representation learning. This framework achieves state-of-the-art performance across four human perception tasks, including DensePose prediction, human parsing, and 3D keypoint prediction. In the area of human reconstruction, we investigate garment reconstruction from 4D point clouds of dressed individuals. To address the ambiguities inherent in 2D images, we propose a principled framework called Garment4D, which enables separable and interpretable garment reconstruction. Notably, Garment4D can effectively reconstruct and model the non-rigid deformations of loose garments (e.g., skirts) that do not share the same topology as the human body. We present two distinct approaches to 3D human generation. In our first work, AvatarCLIP, we generate and animate avatars based on text descriptions of body shapes, appearances, and motions. By leveraging differentiable rendering and large- scale vision-language pre-trained models, we achieve avatar and motion synthesis without the need for supervised training or paired data. To further enhance the quality of the generated avatars, we introduce a second approach, EVA3D, which is a high-quality unconditional 3D human generative model that requires only 2D image collections for training. We design an efficient compositional human NeRF representation to facilitate high-resolution 3D human sampling and rendering, which is employed in adversarial training. Finally, by harnessing the capabilities of large language models (LLMs), we explore 3D human representation learning with a focus on human motion and the integration of perception, reconstruction, and generation. We introduce EgoLM, a versatile framework for understanding egocentric motion using multi-modal data. This framework incorporates rich contextual information from egocentric videos and motion sensors provided by wearable devices. EgoLM unifies various motion learning tasks, including motion understanding from video and motion data, as well as motion tracking and generation from text or sparse sensor input. Additionally, it enables a novel task unique to wearable devices: generating text descriptions from sparse sensor data. By investigating 3D human representation learning from three distinct perspectives, we have attained a comprehensive understanding of humans that spans both high-level semantics and low-level geometry. This thesis offers a cohesive exploration of human-centric 3D representations, contributing significantly to the field of human-centric vision.

The clinical progression of Alzheimer's disease( AD ) can't be accurately evaluated by single modality data alone. Multi-modal data have a good effect on the diagnosis of AD. Clarifying the complementarity between modalities is crucial for the assessment of each stage of AD. Few studies have specifically explored the complementarity between different modalities due to the lack of completely aligned and paired multi-modal data and the limitation of sample size. However, collecting the full set of aligned and paired data is expensive or even impractical. In addition, the limited number of samples poses a great challenge to the robustness of the model. In this paper, different machine learning( ML ) methods were used to explore data complementarity between T1-weighted magnetic resonance imaging ( MRI ), cerebrospinal fluid ( CSF ), and fluorodeoxyglucose-positron emission tomography ( FDG-PET ) modalities. The different modal data of Alzheimer's Neuroimaging Initiative ( ADNI ) and the self-extracted neuroimaging data were experimentally explored. Experiments show that there is obvious complementarity between MRI and CSF. By fusing MRI and CSF data, three binary classification tasks using multi-modal fusion data have achieved varying degrees of improvement. At the same time, we also explored the important features of multi-modal fusion data through SHapley Additive exPlanations ( SHAP ), and found that most important features are supported by relevant literature.

In general, the ‘small sample (n)-large feature ( Open image in new window )’ problem of bioinformatics, image analysis, high throughput molecular screening, astronomy, and other high dimensional applications makes the features highly collinear. In this context, the paper presents a new feature extraction algorithm to address this ‘large Open image in new window small n’ issue associated with multimodal data sets. The proposed algorithm judiciously integrates the concept of both regularization and shrinkage with canonical correlation analysis to extract important features. To deal with the singularity problem, the proposed method increases the diagonal elements of covariance matrices by using regularization parameters, while the off-diagonal elements are decreased by shrinkage coefficients. The concept of hypercuboid equivalence partition matrix of rough hypercuboid approach is used to compute both significance and relevance measures of a feature. The importance of the proposed algorithm over other existing methods is established extensively on real life multimodal omics data set.

Automatic depression classification from multimodal input data is a challenging task. Modern methods use paralinguistic information such as audio and video signals. Using linguistic information such as speech signals and text data for depression classification is a complicated task in deep learning models. Best audio and video features are built to produce a dependable depression classification system. Textual signals related to depression classification are analyzed using text-based content data. Moreover, to increase the achievements of the depression classification system, audio, visual, and text descriptors are used. So, a deep learning-based depression classification model is developed to detect the person with depression from multimodal data. The EEG signal, Speech signal, video, and text are gathered from standard databases. Four stages of feature extraction take place. In the first stage, the features from the decomposed EEG signals are attained by the empirical mode decomposition (EMD) method, and features are extracted by means of linear and nonlinear feature extraction. In the second stage, the spectral features of the speech signals from the Mel-frequency cepstral coefficients (MFCC) are extracted. In the third stage, the facial texture features from the input video are extracted. In the fourth stage of feature extraction, the input text data are pre-processed, and from the pre-processed data, the textual features are extracted by using the Transformer Net. All four sets of features are optimally selected and combined with the optimal weights to get the weighted fused features using the enhanced mountaineering team-based optimization algorithm (EMTOA). The optimal weighted fused features are finally given to the hybrid attention-based dilated network (HADN). The HDAN is developed by combining temporal convolutional network (TCN) with bidirectional long short-term memory (Bi-LSTM). The parameters in the HDAN are optimized with the assistance of the developed EMTOA algorithm. At last, the classified output of depression is obtained from the HDAN. The efficiency of the developed deep learning HDAN is validated by comparing it with various traditional classification models.

More and more healthcare data are becoming readily available nowadays. These data can help the healthcare professionals and patient themselves to better understand the patient status and potentially lead to improved care quality. However, the analysis of these data are challenging because they are large-scale and heterogeneous, high-dimensional and sparse, temporal but irregularly sampled. In this paper, we propose a method called Double Core Memory Networks (DCMN) to integrate information from different modalities of the longitudinal patient data and learn a joint patient representation effective for downstream analytical tasks such as risk prediction. DCMN is designed not only to disentangle the temporal and non-linear intra-modal dependencies for the data within each modality but also to capture the long-term inter-modal interactions. DCMN models are the end-to-end memory networks with two external memory cores where each modality of data is compressed and stored. Each memory core has an information-flow controller named query to interact with an external memory module. In addition, we incorporate a gating mechanism into basic DCMN model to perform dynamic regulation of memory interaction. DCMN models have multiple computational layers (hops) allowing data of different modalities interacting with each other recurrently along with a mechanism of alternating access of external memory for each memory core hop-by-hop. We evaluate DCMN models on two outcome prediction tasks, including a mortality prediction on the public Medical Information Mart for Intensive Care III (MIMIC-III) database and a cost prediction on the Hospital Quality Monitoring System (HQMS) dataset. Experimental results demonstrate that our DCMN models are more competitive over the baseline methods in the multimodal prediction setting.

Recent developments of video and sensing technology have led to large volumes of digital media data. Current media production relies on videos from the principal camera together with a wide variety of heterogeneous source of supporting data [photos, light detection and ranging point clouds, witness video camera, high dynamic range imaging, and depth imagery]. Registration of visual data acquired from various 2D and 3D sensing modalities is challenging because current matching and registration methods are not appropriate due to differences in structure, format, and noise characteristics for multimodal data. A combined 2D/3D visualization of this registered data allows an integrated overview of the entire data set. For such a visualization, a Web-based context presents several advantages. In this paper, we propose a unified framework for registration and visualization of this type of visual media data. A new feature description and matching method is proposed, adaptively considering local geometry, semiglobal geometry, and color information in the scene for more robust registration. The resulting registered 2D/3D multimodal visual data are too large to be downloaded and viewed directly via the Web browser, while maintaining an acceptable user experience. Thus, we employ hierarchical techniques for compression and restructuring to enable efficient transmission and visualization over the Web, leading to interactive visualization as registered point clouds, 2D images, and videos in the browser, improving on the current state-of-the-art techniques for Web-based visualization of big media data. This is the first unified 3D Web-based visualization of multimodal visual media production data sets. The proposed pipeline is tested on big multimodal data set typical of film and broadcast production, which are made publicly available. The<br/>proposed feature description method shows two times higher precision of feature matching and more stable registration performance than existing 3D feature descriptors.

This paper presents a novel framework integrating the RePaint method with the performance-guided denoising diffusion probabilistic model (DDPM) to complete missing or undefined design components based on any given partial structures while satisfying specified performance targets. RePaint allows a pre-trained diffusion model to synthesize designs under the condition of partial designs, offering flexibility and controllability of generative design results. Using a dataset of parametric ship hull designs, a performance-guided diffusion model is pre-trained to generate designs with targeted total resistance coefficients. Experiment results demonstrate that our method generates new designs from random incomplete ship hull designs with performances close to those generated by the pre-trained models. The study further analyzes factors that might influence the new framework’s performance and reveals that the pre-trained model’s design parameter value distribution has a significant effect on the new method’s output designs. This distribution can be leveraged to guide the synthesis of new designs with desirable performance.

This research presents a locality-sensitive hashing framework that enhances approximate nearest neighbor search efficiency by integrating adaptive encoding trees and BERT-based clusterization. The proposed method optimizes data space partitioning before applying hashing, improving retrieval accuracy while reducing computational complexity. First, multimodal data, such as images and textual descriptions, are transformed into a unified semantic space using pre-trained bidirectional encoder representations from transformers embeddings. this ensures cross-modal consistency and facilitates high-dimensional similarity comparisons. Second, dimensionality reduction techniques like Uniform Manifold Approximation and Projection or t-distributed stochastic neighbor embedding are applied to mitigate the curse of dimensionality while preserving key relationships between data points. Third, an adaptive encoding tree locality-sensitive hashing encoding tree is constructed, dynamically segmenting the data space based on statistical distribution, thereby enabling efficient hierarchical clustering. Each data point is converted into a symbolic representation, allowing fast retrieval using structured hashing. Fourth, locality-sensitive hashing is applied to the encoded dataset, leveraging p-stable distributions to maintain high search precision while reducing index size. The combination of encoding trees and Locality-Sensitive Hashing enables efficient candidate selection while minimizing search overhead. Experimental evaluations on the CarDD dataset, which includes car damage images and annotations, demonstrate that the proposed method outperforms state-of-the-art approximate nearest neighbor techniques in both indexing efficiency and retrieval accuracy. The results highlight its adaptability to large-scale, high-dimensional, and multimodal datasets, making it suitable for diagnostic models and real-time retrieval tasks.

Accurately estimating urban floodwater depth is a critical step in enhancing urban resilience and strengthening disaster prevention and mitigation capabilities. Traditional methods relying on hydrological monitoring stations and numerical simulations suffer from limitations such as sparse spatial coverage, insufficient validation data, limited accuracy, and delayed fast performance. In contrast, social media data—characterized by its vast volume and fast availability, can effectively compensate for these shortcomings. When processed using artificial intelligence (AI) algorithms, such data can significantly improve credibility, disaster perception speed, and water depth estimation accuracy. To address these challenges, this paper proposes a robust and widely applicable method for rapid urban flood depth perception. The approach integrates AI technology and social media data to construct an AI framework capable of perceiving urban physical parameters through multimodal big data fusion without costly model training. By leveraging the near real-time and widespread nature of social media, an automated web crawler collects flood images and their textual descriptions (including reference objects), eliminating the need for additional hardware investments. The framework uses predefined prompts and pre-trained models to automatically perform relevance verification, duplicate filtering, object detection, and feature extraction, requiring no manual data annotation or model training. With only a minimal amount of water depth annotated data and compressed cross-modal feature vectors as training input, a lightweight Multilayer Perceptron (MLP) achieves high-precision depth estimation based on reference objects. This method avoids the need for large-scale model fine-tuning, allowing rapid training even on devices without GPUs. Experiments demonstrate that the proposed method reduces the Mean Square Error (MSE) by over 80%, processes each image in less than 0.5 s (more than 20 times faster than existing large-model approaches), and exhibits strong robustness to changes in perspective and image quality. The solution is fully compatible with existing infrastructure such as surveillance cameras, offering an efficient and reliable approach for fast flood monitoring in urban hydrology and water engineering applications.

Due to the recent success of diffusion models, text-to-image generation is becoming increasingly popular and achieves a wide range of applications. Among them, text-to-image editing, or continuous text-to-image generation, attracts lots of attention and can potentially improve the quality of generated images. It's common to see that users may want to slightly edit the generated image by making minor modifications to their input textual descriptions for several rounds of diffusion inference. However, such an image editing process suffers from the low inference efficiency of many existing diffusion models even using GPU accelerators. To solve this problem, we introduce Fast Image Semantically Edit (FISEdit), a cached-enabled sparse diffusion model inference engine for efficient text-to-image editing. The key intuition behind our approach is to utilize the semantic mapping between the minor modifications on the input text and the affected regions on the output image. For each text editing step, FISEdit can 1) automatically identify the affected image regions and 2) utilize the cached unchanged regions' feature map to accelerate the inference process. For the former, we measure the differences between cached and ad hoc feature maps given the modified textual description, extract the region with significant differences, and capture the affected region by masks. For the latter, we develop an efficient sparse diffusion inference engine that only computes the feature maps for the affected region while reusing the cached statistics for the rest of the image. Finally, extensive empirical results show that FISEdit can be 3.4 times and 4.4 times faster than existing methods on NVIDIA TITAN RTX and A100 GPUs respectively, and even generates more satisfactory images.

This systematic review on data modalities synthesises the research findings in terms of how to optimally use and combine such modalities when investigating cognitive, motivational, and emotional learning processes. ERIC, WoS, and ScienceDirect databases were searched with specific keywords and inclusion criteria for research on data modalities, resulting in 207 relevant publications. We provide findings in terms of target journal, country, subject, participant characteristics, educational level, foci, type of data modality, research method, type of learning, learning setting, and modalities used to study the different foci. In total, 18 data modalities were classified. For the 207 multimodal publications, 721 occurrences of modalities were observed. The most popular modality was interview followed by survey and observation. The least common modalities were heart rate variability, facial expression recognition, and screen recording. From the 207 publications, 98 focused exclusively on the cognitive aspects of learning, followed by 27 publications that only focused on motivation, while only five publications exclusively focused on emotional aspects. Only 10 publications focused on a combination of cognitive, motivational, and emotional aspects of learning. Our results plea for the increased use of objective measures, highlight the need for triangulation of objective and subjective data, and demand for more research on combining various aspects of learning. Further, rather than researching cognitive, motivational, and emotional aspects of learning separately, we encourage scholars to tap into multiple learning processes with multimodal data to derive a more comprehensive view on the phenomenon of learning.

The Big Data era in Biomedical research has resulted in large-cohort data repositories such as The Cancer Genome Atlas (TCGA). These repositories routinely contain hundreds of matched patient samples for genomic, proteomic, imaging, and clinical data modalities, enabling holistic and multi-modal integrative analysis of human disease. Using TCGA renal and ovarian cancer data, we conducted a novel investigation of multi-modal data integration by combining histopathological image and RNA-seq data. We compared the performances of two integrative prediction methods: majority vote and stacked generalization. Results indicate that integration of multiple data modalities improves prediction of cancer grade and outcome. Specifically, stacked generalization, a method that integrates multiple data modalities to produce a single prediction result, outperforms both single-data-modality prediction and majority vote. Moreover, stacked generalization reveals the contribution of each data modality (and specific features within each data modality) to the final prediction result and may provide biological insights to explain prediction performance.

The rich tapestry of cultural diversity is intricately woven by artists who have traversed various cultural landscapes, often producing distinctive artistic expressions. Among these artists, African American female painters stand out with their unmistakable styles and themes, a testament to their unique narrative. To explore the elements that shape the work of these artists in the United States, an extensive review of literature was conducted, distilling the essence of their creative output. The analysis encompasses personal and societal dimensions, unveiling a tapestry of influences including personal experiences, societal encounters, artistic styles, environmental nuances, technological advancements, sources of inspiration, and collaborative relationships. Through meticulous research and surveying, this investigation identified the paramount features that resonate most profoundly with the artistic vision of eight eminent painters. This study highlights the top three as being most influential in their creative journey. Future research endeavors should focus on conducting a more systematic case study of African American artists in order to provide a comprehensive analysis of their features. This will involve finding primary information and reference materials to make a deeper investigation into the influencing features on Africa American female artists between the 1960s and 2000s.

GeoPredict-LLM: Intelligent tunnel advanced geological prediction by reprogramming large language models