Embracing Multimodal Data in Multimedia Data Analysis

The value of artificial intelligence combined with multimodal data analysis in tumor immunotherapy and targeted therapy.

Single-cell omics enable the dissection of cellular heterogeneity, yet the high dimensionality, inherent noise, and sparsity present significant challenges. These challenges are amplified for rare cell populations, which are often difficult to annotate reliably but can be central to development and disease. As single-cell assays increasingly capture multiple molecular layers, the integrative analysis of such multimodal data further increases complexity. Here, we propose Hydra, a deep generative framework based on an ensemble of variational autoencoders for effective learning of unimodal and multimodal single-cell omics data. Hydra implements interpretable modules for capturing cell-type-specific molecular signatures. The ensemble of such interpretable modules enables reproducible feature selection and robust cell-type annotation, with particular effectiveness for rare populations. We benchmarked Hydra on a repertoire of 21 datasets, including unimodal and multimodal single-cell omics data. Our results demonstrate that Hydra offers comparable to superior performance to several state-of-the-art methods. Finally, we highlight the utility of Hydra in robustly annotating brain cellular subtypes and preserving disease-relevant signatures using our previously published dataset that profiles Alzheimer's disease.

Scars that form after skin injury can cause structural and functional skin damage. Currently, scar tissue determination relies mainly on doctors’ subjective observations and judgments and lacks objectivity. However, current deep learning models can only achieve specific discrimination using unimodal data, which limits the comprehensive understanding of scar tissue and may reduce accuracy and stability. To solve these problems, in this study, a skin scar recognition platform based on advanced deep learning and a weighted aggregation network fusion method is proposed. It is implemented using a residual network-based CNN model and a logistic regression model with L1 regularization and is suitable for both unimodal and multimodal data. The experimental results showed that the proposed platform achieved a satisfactory accuracy of 98.26% for image discrimination. In the gene discrimination model test performed on a test dataset containing 17 gene expression samples, all samples were accurately discriminated. In addition, the proposed multimodal discrimination model achieved a discrimination accuracy of 98.23%. These results validate the effectiveness of deep feature extraction and multimodal feature fusion techniques for image discrimination tasks. On this basis, to deeply explore the pathogenesis of scar formation, a method with the ability to integrate regularization, sparsity, and orthogonality constraints, multiconstraint joint non-negative matrix factorization (MCJNMF), was used to explore the genetic correlation between collagen micrographic image features and gene expression data. In this study, we confirmed the association between the calcium signaling pathway, MAPK signaling pathway, and collagen fiber repair, and successfully identified 11 potential therapeutic targets, including TRIM59 and TBC1D9, which provide important clues for future scar treatment and prevention strategies.

In general, the ‘small sample (n)-large feature ( Open image in new window )’ problem of bioinformatics, image analysis, high throughput molecular screening, astronomy, and other high dimensional applications makes the features highly collinear. In this context, the paper presents a new feature extraction algorithm to address this ‘large Open image in new window small n’ issue associated with multimodal data sets. The proposed algorithm judiciously integrates the concept of both regularization and shrinkage with canonical correlation analysis to extract important features. To deal with the singularity problem, the proposed method increases the diagonal elements of covariance matrices by using regularization parameters, while the off-diagonal elements are decreased by shrinkage coefficients. The concept of hypercuboid equivalence partition matrix of rough hypercuboid approach is used to compute both significance and relevance measures of a feature. The importance of the proposed algorithm over other existing methods is established extensively on real life multimodal omics data set.

Research on multimodal data analysis such as annotated image analysis is becoming more important than ever due to the increase in the amount of data. One of the approaches to this problem is multimodal topic models as an extension of latent Dirichlet allocation (LDA). Symmetric correspondence topic models (SymCorrLDA) are state-of-the-art multimodal topic models that can appropriately model multimodal data considering inter-modal dependencies. Incidentally, hierarchically structured categories can help users find relevant data from a large amount of data collection. Hierarchical topic models such as hierarchical latent Dirichlet allocation (hLDA) can discover a tree-structured hierarchy of latent topics from a given unimodal data collection; however, no hierarchical topic models can appropriately handle multimodal data considering intermodal mutual dependencies. In this paper, we propose h-SymCorrLDA to discover latent topic hierarchies from multimodal data by combining the ideas of the two previously mentioned models: multimodal topic models and hierarchical topic models. We demonstrate the effectiveness of our model compared with several baseline models through experiments with two datasets of annotated images.

Multimodal single-cell omics technologies enable multiple molecular programs to be simultaneously profiled at a global scale in individual cells, creating opportunities to study biological systems at a resolution that was previously inaccessible. However, the analysis of multimodal single-cell omics data is challenging due to the lack of methods that can integrate across multiple data modalities generated from such technologies. Here, we present Matilda, a multi-task learning method for integrative analysis of multimodal single-cell omics data. By leveraging the interrelationship among tasks, Matilda learns to perform data simulation, dimension reduction, cell type classification, and feature selection in a single unified framework. We compare Matilda with other state-of-the-art methods on datasets generated from some of the most popular multimodal single-cell omics technologies. Our results demonstrate the utility of Matilda for addressing multiple key tasks on integrative multimodal single-cell omics data analysis. Matilda is implemented in Pytorch and is freely available from https://github.com/PYangLab/Matilda.

The low birth rate society has had a profound impact on cultural consumption patterns and product design requirements. The design of cultural creative (cultural-creative) products faces new challenges in meeting the diverse emotional needs of users. This study aims to explore the emotional demands and preference characteristics of different generational users for cultural creative product design in a low birth rate society, as well as how design optimization can better fulfill these needs. A multimodal data collection and analysis approach integrating EEG (electroencephalography) and behavioral experiments was employed. EEG technology was utilized to record users' immediate emotional responses to design elements in cultural creative products, while behavioral experiments observed users' interaction behaviors to validate their actual demands. The research design involved multimodal data collection in a controlled laboratory setting, coupled with data fusion and analysis using deep learning and data mining techniques. This approach enabled precise profiling of users' emotional responses and design preferences, providing a scientific foundation for the innovation and optimization of cultural creative products. By extending the application scenarios of multimodal data and fostering interdisciplinary research, this study offers new momentum for the sustainable development of the cultural creative industry in the context of a low birth rate society.

Multi-modal big data analysis has emerged as a vital tool within the Industrial Internet of Things (IIoT). Despite its significance, challenges persist, particularly the absence of an efficient analytical architecture. In this study, we introduce a novel multi-modal big data analysis framework tailored for IIoT. Our approach features a cloud-edge collaborative training solution designed to optimize the training of deep learning models, along with an edge-based rapid reasoning solution for deploying models and facilitating swift inference on resource-constrained devices in industrial settings.

Multimedia systems, such as social media platforms, play a crucial role in disseminating vital information during calamities. This information is shared in various formats such as images, text, videos, audio, etc. Therefore, it becomes important to have a system that can identify multimodal data to classify relevant information. This paper proposes a new age classification method for multimodal data using advanced and improved transformer models, such as Vision Transformer and Generative Pre-trained Transformer 2, for image and text classification, respectively. These models were combined using an ensemble model (Random Forest Classifier), achieving an accuracy of 84.66% on the multimodal data. Furthermore, the proposed model demonstrates higher prediction accuracy compared to traditional Convolutional Neural Network (CNN) models which have an accuracy of 71.43%, exceeding it by 13.23%. A comparison with convolutional models is conducted to underscore the advantages of transformer models and to substantiate the necessity of the experiment. Our proposed classification model using Vision Transformer and GPT-2, along with an ensemble model, can be replicated by researchers in disaster management, humanitarian aid organizations, and social media platforms looking to filter and prioritize information during emergencies.

This paper pioneers a novel exploration of environmental impacts in livestock farming, focusing on pig farming’s intersection with climate change and sustainability. It emphasizes the transformative potential of data-driven Artificial Intelligence (AI) methodologies, specifically the Internet of Things (IoT) and multimodal data analysis, in promoting equitable and sustainable food systems. The study observes five pigs aged 86 to 108 days using a tripartite sensor that records heart rate, respiration rate, and accelerometer data. The unique experimental design alternates between periods of isolation during feeding and subsequent pairing, enabling the investigation of stress-induced changes. Key inquiries include discerning patterns in heart rate data during isolation versus paired settings, fluctuations in respiration rates, and behavioral shifts induced by isolation or pairing. The study also explores the potential detection of gait abnormalities, correlations between pigs’ age and their gait or activity patterns, and the evolution of pigs’ walking abilities with age. The paper scrutinizes accelerometer data to detect activity changes when pigs are paired, potentially indicating increased stress or aggression. It also examines the adaptation of pigs to alternating isolation and pairing over time and how their heart rate, respiration rate, and activity data reflect this process. The study considers other significant variables, such as time of day and isolation duration, affecting the pigs’ physiological parameters. Sensor data are further utilized to identify behavioral patterns during periods of feeding, isolation, or pairing. In conclusion, this study harnesses IoT and multimodal data analysis in a groundbreaking approach to pig welfare research. It underscores the compelling potential of technology to inform about overall pig welfare, particularly stress levels and gait quality, and the power of data-driven insights in fostering equitable, healthy, and environmentally conscious livestock production systems.

In smart homes, data generated from real-time sensors for human activity recognition is complex, noisy and imbalanced. It is a significant challenge to create machine learning models that can classify activities which are not as commonly occurring as other activities. Machine learning models designed to classify imbalanced data are biased towards learning the more commonly occurring classes. Such learning bias occurs naturally, since the models better learn classes which contain more records. This paper examines whether fusing real-world imbalanced multi-modal sensor data improves classification results as opposed to using unimodal data; and compares deep learning approaches to dealing with imbalanced multi-modal sensor data when using various resampling methods and deep learning models. Experiments were carried out using a large multi-modal sensor dataset generated from the Sensor Platform for HEalthcare in a Residential Environment (SPHERE). The data comprises 16104 samples, where each sample comprises 5608 features and belongs to one of 20 activities (classes). Experimental results using SPHERE demonstrate the challenges of dealing with imbalanced multi-modal data and highlight the importance of having a suitable number of samples within each class for sufficiently training and testing deep learning models. Furthermore, the results revealed that when fusing the data and using the Synthetic Minority Oversampling Technique (SMOTE) to correct class imbalance, CNN-LSTM achieved the highest classification accuracy of 93.67% followed by CNN, 93.55%, and LSTM, i.e. 92.98%.

The volume of data in the world is growing very fast and generated from verity of sources like social media, sensors airline industry or scientific data in different formats. Biggest challenge is how to infer meaningful insights from such a varietyful and big data along with concern of data storage and management of fast growing data. The size of the databases used in today’s enterprises has been growing at exponential rates day by day. Hence, industries requirement to quickly process and analyze the big data volumes for business decision making and customer insights has also grown exponentially. Data pouring from various sources may be can be structured or unstructured in nature. Structured data refers to a relatively well-organized information, which can be further inserted into traditional RDBMS. As Traditional RDBMS are efficient and easy queries by simple, straightforward search algorithms or SQL queries. In contrast to structured data, unstructured data can be considered as information, which does not, comes in a pre-defined data format, well organized data storage model, or cannot be stored well into relational tables. It is assumed to be fastest growing type of data, e.g. image, sensors data, web chats, social networking messaging data, video, documents, log files, and email data. There are many techniques and software available, which can process and provide efficient storage of unstructured data and help organization to perform analytics on unstructured data. Unstructured data does not well-organized and not stored in predefined manner e.g. logs, web chats. The variety and on ordered nature of data makes storage methods and structure makes execution a time and resource-consuming affair. Advancement into technology has open floodgates to push huge volume of unstructured type of data. Multimedia data is one of the example of unstructured big data, which spans all over the Internet. This needs high execution capability to extract useful information. Rapid processing of multimedia data such as video is important for e.g. criminal investigations, surveillance monitoring, news analysis, sports analytics domain, emotion extraction, etc. Hence, analysis of multimedia data in minimum timeframe is one of the latest research areas. Therefore, we have researched techniques for analyzing unstructured data to extract meaningful information hidden in the big data. In addition, we will describe about various techniques and software used to Manage, process unstructured big data in efficient manner, and increases the performance of complexity analysis.

Multi-modal advanced deep learning architectures for breast cancer survival prediction

In the era of big data, a large number of multimodal data are constantly generated and accumulated. In the practical application analysis of multimodal big data, the modal incompleteness, real-time processing, modal imbalance and high-dimensional attributes pose severe challenges to the multimodal fusion method. In view of the above characteristics of multimodal data and the shortcomings of existing multimodal fusion algorithms, this chapter studies the multimodal data fusion algorithms from four aspects: incomplete modal analysis fusion, incremental modal clustering fusion, heterogeneous modal migration fusion and low dimensional modal sharing fusion. A series of multimodal data fusion models and algorithms are designed.KeywordsMultimodal dataData fusionUnsupervised learning

Advances in deep learning and representation learning have transformed item factor analysis (IFA) in the item response theory (IRT) literature by enabling more efficient and accurate parameter estimation. Variational autoencoders (VAEs) are widely used to model high-dimensional latent variables in this context, but the limited expressiveness of their inference networks can still hinder performance. We introduce adversarial variational Bayes (AVB) and an importance-weighted extension (IWAVB) as more flexible inference algorithms for IFA. By combining VAEs with generative adversarial networks (GANs), AVB uses an auxiliary discriminator network to frame estimation as a two-player game and removes the restrictive standard normal assumption on the latent variables. Theoretically, AVB and IWAVB can achieve likelihoods that match or exceed those of VAEs and importance-weighted autoencoders (IWAEs). In exploratory analyses of empirical data, IWAVB attained higher likelihoods than IWAE, indicating greater expressiveness. In confirmatory simulations, IWAVB achieved comparable mean-square error in parameter recovery while consistently yielding higher likelihoods, and it clearly outperformed IWAE when the latent distribution was multimodal. These findings suggest that IWAVB can scale IFA to complex, large-scale, and potentially multimodal settings, supporting closer integration of psychometrics with modern multimodal data analysis.