Multimodal CustOmics: A unified and interpretable multi-task deep learning framework for multimodal integrative data analysis in oncology.

It is an essential task to accurately diagnose cancer subtypes in computational pathology for personalized cancer treatment. Recent studies have indicated that the combination of multimodal data, such as whole slide images (WSIs) and multi-omics data, could achieve more accurate diagnosis. However, robust cancer diagnosis remains challenging due to the heterogeneity among multimodal data, as well as the performance degradation caused by insufficient multimodal patient data. In this work, we propose a novel multimodal co-attention fusion network (MCFN) with online data augmentation (ODA) for cancer subtype classification. Specifically, a multimodal mutual-guided co-attention (MMC) module is proposed to effectively perform dense multimodal interactions. It enables multimodal data to mutually guide and calibrate each other during the integration process to alleviate inter- and intra-modal heterogeneities. Subsequently, a self-normalizing network (SNN)-Mixer is developed to allow information communication among different omics data and alleviate the high-dimensional small-sample size problem in multi-omics data. Most importantly, to compensate for insufficient multimodal samples for model training, we propose an ODA module in MCFN. The ODA module leverages the multimodal knowledge to guide the data augmentations of WSIs and maximize the data diversity during model training. Extensive experiments are conducted on the public TCGA dataset. The experimental results demonstrate that the proposed MCFN outperforms all the compared algorithms, suggesting its effectiveness.

Enhancing Personalized Prognostic Assessment of Myelodysplastic Syndromes through a Multimodal and Explainable Deep Data Fusion Approach (MAGAERA)

The value of artificial intelligence combined with multimodal data analysis in tumor immunotherapy and targeted therapy.

Single-cell omics enable the dissection of cellular heterogeneity, yet the high dimensionality, inherent noise, and sparsity present significant challenges. These challenges are amplified for rare cell populations, which are often difficult to annotate reliably but can be central to development and disease. As single-cell assays increasingly capture multiple molecular layers, the integrative analysis of such multimodal data further increases complexity. Here, we propose Hydra, a deep generative framework based on an ensemble of variational autoencoders for effective learning of unimodal and multimodal single-cell omics data. Hydra implements interpretable modules for capturing cell-type-specific molecular signatures. The ensemble of such interpretable modules enables reproducible feature selection and robust cell-type annotation, with particular effectiveness for rare populations. We benchmarked Hydra on a repertoire of 21 datasets, including unimodal and multimodal single-cell omics data. Our results demonstrate that Hydra offers comparable to superior performance to several state-of-the-art methods. Finally, we highlight the utility of Hydra in robustly annotating brain cellular subtypes and preserving disease-relevant signatures using our previously published dataset that profiles Alzheimer's disease.

Multimedia systems, such as social media platforms, play a crucial role in disseminating vital information during calamities. This information is shared in various formats such as images, text, videos, audio, etc. Therefore, it becomes important to have a system that can identify multimodal data to classify relevant information. This paper proposes a new age classification method for multimodal data using advanced and improved transformer models, such as Vision Transformer and Generative Pre-trained Transformer 2, for image and text classification, respectively. These models were combined using an ensemble model (Random Forest Classifier), achieving an accuracy of 84.66% on the multimodal data. Furthermore, the proposed model demonstrates higher prediction accuracy compared to traditional Convolutional Neural Network (CNN) models which have an accuracy of 71.43%, exceeding it by 13.23%. A comparison with convolutional models is conducted to underscore the advantages of transformer models and to substantiate the necessity of the experiment. Our proposed classification model using Vision Transformer and GPT-2, along with an ensemble model, can be replicated by researchers in disaster management, humanitarian aid organizations, and social media platforms looking to filter and prioritize information during emergencies.

In general, the ‘small sample (n)-large feature ( Open image in new window )’ problem of bioinformatics, image analysis, high throughput molecular screening, astronomy, and other high dimensional applications makes the features highly collinear. In this context, the paper presents a new feature extraction algorithm to address this ‘large Open image in new window small n’ issue associated with multimodal data sets. The proposed algorithm judiciously integrates the concept of both regularization and shrinkage with canonical correlation analysis to extract important features. To deal with the singularity problem, the proposed method increases the diagonal elements of covariance matrices by using regularization parameters, while the off-diagonal elements are decreased by shrinkage coefficients. The concept of hypercuboid equivalence partition matrix of rough hypercuboid approach is used to compute both significance and relevance measures of a feature. The importance of the proposed algorithm over other existing methods is established extensively on real life multimodal omics data set.

Multimedia data analysis is at the heart of multimedia research, where the patterns, information, and knowledge are mined and recognized from the multimedia data for various applications. We have witnessed significant performance improvements in multimedia data analysis driven by the recent advances in artificial intelligence and deep learning. While most of the existing methods focus on analyzing unimodal data, many novel multimodal multimedia data analysis techniques have also been developed and achieved better performance than those using unimodal data. Multimodal multimedia data has proven to be effective to improve the performance and reliability of data analysis systems and applications. It is also important to match and retrieve data with similar contents across modalities. However, due to the highly complex nature of multimodal multimedia data, further research in multimodal multimedia data analysis is needed to address the challenges including missing modalities, lack of benchmark datasets, and so on.

Invasive lobular breast cancer (ILC) is an understudied subtype of breast cancer with late recurrence, metastasis to serosal surfaces, such as the peritoneum, and dismal long-term outcome. We previously reported better patient outcome in ER+ breast cancers with high glucocorticoid receptor (GR) expression (Pan et al 2011). Preliminary data suggests more favorable patient outlook in the ILC subtype when tumors express higher levels of GR. The dynamic interaction between a tumor and its microenvironment leads to phenotypic changes in stromal cells and the extracellular matrix to promote growth or invasion of malignant cells. ILC is histologically distinct from invasive ductal carcinoma and is characterized by discohesive tumor cells that grow as “Indian files” due to lack of the cell adhesion molecule E-cadherin. Because of these differences we expect the tumor microenvironment (TME) to be quite unique in ILC compared to other breast cancer subtypes. We hypothesized that differing levels of expression of nuclear receptors by tumor cells would impact cells residing within the stroma, presumably through paracrine signaling. Immunohistochemistry for 25 ILC biopsies revealed a wide variation in GR expression, ranging from completely negative to strongly positive. Our goal was to gain insight into how GR presence or absence in ILC cells might impact gene and protein expression in malignant cells and their TME. We were inspired to examine how crosstalk between the GR-positive or GR-negative cancer cells and their respective tumor microenvironment (TME) differentially impact stromal cell gene expression as well as the immune cell milieu. Bulk expression profiling is inadequate to examine molecular profiles of the tumor and stroma separately. Therefore, we used nanoString GeoMx® high-plex digital spatial profiling of RNA and protein expression in GR-positive and GR-negative primary ILC. We first performed spatial gene expression using the Whole Transcriptome Atlas (WTA) for six primary ILC, two of which were strongly positive for GR, two GR-negative and 2 tumors harboring both GR-positive and GR-negative regions. To examine the tumor cells and the stroma independently we segmented regions into panCK-positive and panCK-negative segments and analyzed those separately. Intriguing differences were observed between GR-positive and negative ILC, for the tumor cells as well as for the TME. Implementation of the Spatial Deconvolution script embedded within the GeoMx® software revealed striking differences in the abundance of immune cells, endothelial cells and fibroblasts in the stroma of GR-positive vs GR-negative ILC. Macrophages and other myeloid subsets were present in significantly higher numbers in the TME of GR-positive compared to GR-negative ILC. The contribution of B- and T-cell subsets as well as endothelial cells and fibroblasts were also notably different. To more precisely define the spatial distribution and abundance of various cell subtypes and their association with tumor cells, we are including protein DSP as an added layer to complement the WTA. We are also performing RNA and protein DSP for an additional ten GR-positive and ten GR-negative ILC. Thus, we will test our initial observations and expand our investigation to generate comprehensive molecular profiles of ILC tumors and stromal cells and acquire a deeper understanding of how GR expression/activation influences the crosstalk between ILC cells and their TME. We anticipate that differences in gene and protein expression will provide clues as to how GR activation affects proliferative and adhesive properties of ILC through modification of the TME. Citation Format: Lynda B. Bennett, Candace Frerich, Sunati Sahoo, Cheryl Lewis, Indu Raman, Min Xu, Guanchun Chen, Suzanne D. Conzen. Digital spatial profiling of RNA and protein in the invasive lobular breast cancer microenvironment [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P2-21-08.

In the era of big data, a large number of multimodal data are constantly generated and accumulated. In the practical application analysis of multimodal big data, the modal incompleteness, real-time processing, modal imbalance and high-dimensional attributes pose severe challenges to the multimodal fusion method. In view of the above characteristics of multimodal data and the shortcomings of existing multimodal fusion algorithms, this chapter studies the multimodal data fusion algorithms from four aspects: incomplete modal analysis fusion, incremental modal clustering fusion, heterogeneous modal migration fusion and low dimensional modal sharing fusion. A series of multimodal data fusion models and algorithms are designed.KeywordsMultimodal dataData fusionUnsupervised learning

Research on multimodal data analysis such as annotated image analysis is becoming more important than ever due to the increase in the amount of data. One of the approaches to this problem is multimodal topic models as an extension of latent Dirichlet allocation (LDA). Symmetric correspondence topic models (SymCorrLDA) are state-of-the-art multimodal topic models that can appropriately model multimodal data considering inter-modal dependencies. Incidentally, hierarchically structured categories can help users find relevant data from a large amount of data collection. Hierarchical topic models such as hierarchical latent Dirichlet allocation (hLDA) can discover a tree-structured hierarchy of latent topics from a given unimodal data collection; however, no hierarchical topic models can appropriately handle multimodal data considering intermodal mutual dependencies. In this paper, we propose h-SymCorrLDA to discover latent topic hierarchies from multimodal data by combining the ideas of the two previously mentioned models: multimodal topic models and hierarchical topic models. We demonstrate the effectiveness of our model compared with several baseline models through experiments with two datasets of annotated images.

Multi-modal big data analysis has emerged as a vital tool within the Industrial Internet of Things (IIoT). Despite its significance, challenges persist, particularly the absence of an efficient analytical architecture. In this study, we introduce a novel multi-modal big data analysis framework tailored for IIoT. Our approach features a cloud-edge collaborative training solution designed to optimize the training of deep learning models, along with an edge-based rapid reasoning solution for deploying models and facilitating swift inference on resource-constrained devices in industrial settings.

The multiomics data are heterogeneous and come from different biological levels such as epigenetics, genomics, transcriptomics and proteomics. The development of high-throughput technologies has enabled researchers not only to study all the entities together but also to utilize information from different levels spanning DNA methylation, copy number variation (CNV), mutation, gene expression, and miRNA expression. With the recent advancement in image informatics, the field of radiomics is rapidly emerging. It can be expected that the information from microscopic images of the tissue will soon be part of many multiomics studies. Meanwhile, integration of different kinds of multiomics data to extract relevant biological information is currently a big challenge. This study is our ongoing effort to develop a model that properly integrates multiomics data and allows easy retrieval of information relevant to biological processes. In this article, we have enriched our previous graph database model to store gene expression, miRNA expression, DNA methylation, mutation, CNV, clinical data, including information of the image of tissue slides. To show that the model is working, we used data from the Cancer Genome Atlas for three cancer types.

Multimodal single-cell omics technologies enable multiple molecular programs to be simultaneously profiled at a global scale in individual cells, creating opportunities to study biological systems at a resolution that was previously inaccessible. However, the analysis of multimodal single-cell omics data is challenging due to the lack of methods that can integrate across multiple data modalities generated from such technologies. Here, we present Matilda, a multi-task learning method for integrative analysis of multimodal single-cell omics data. By leveraging the interrelationship among tasks, Matilda learns to perform data simulation, dimension reduction, cell type classification, and feature selection in a single unified framework. We compare Matilda with other state-of-the-art methods on datasets generated from some of the most popular multimodal single-cell omics technologies. Our results demonstrate the utility of Matilda for addressing multiple key tasks on integrative multimodal single-cell omics data analysis. Matilda is implemented in Pytorch and is freely available from https://github.com/PYangLab/Matilda.

BackgroundLung cancer has become the most common cancer type and caused the most cancer deaths. Lung adenocarcinoma (LUAD) is one of the major types of lung cancer. Accumulating evidence suggests the tumor microenvironment is correlated with the tumor progress and the patient’s outcome. This study aimed to establish a gene signature based on tumor microenvironment that can predict patients’ outcomes for LUAD.MethodsDataset TCGA-LUAD, downloaded from the TCGA portal, were taken as training cohort, and dataset GSE72094, obtained from the GEO database, was set as validation cohort. In the training cohort, ESTIMATE algorithm was applied to find intersection differentially expressed genes (DEGs) among tumor microenvironment. Kaplan–Meier analysis and univariate Cox regression model were performed on intersection DEGs to preliminarily screen prognostic genes. Besides, the LASSO Cox regression model was implemented to build a multi-gene signature, which was then validated in the validation cohorts through Kaplan–Meier, Cox, and receiver operating characteristic curve (ROC) analyses. In addition, the correlation between tumor mutational burden (TMB) and risk score was evaluated by Spearman test. GSEA and immune infiltrating analyses were conducted for understanding function annotation and the role of the signature in the tumor microenvironment.ResultsAn eight-gene signature was built, and it was examined by Kaplan–Meier analysis, revealing that a significant overall survival difference was seen. The eight-gene signature was further proven to be independent of other clinico-pathologic parameters via the Cox regression analyses. Moreover, the ROC analysis demonstrated that this signature owned a better predictive power of LUAD prognosis. The eight-gene signature was correlated with TMB. Furthermore, GSEA and immune infiltrating analyses showed that the exact pathways related to the characteristics of eight-genes signature, and identified the vital roles of Mast cells resting and B cells naive in the prognosis of the eight-gene signature.ConclusionIdentifying the eight-gene signature (INSL4, SCN7A, STAP1, P2RX1, IKZF3, MS4A1, KLRB1, and ACSM5) could accurately identify patients’ prognosis and had close interactions with Mast cells resting and B cells naive, which may provide insight into personalized prognosis prediction and new therapies for LUAD patients.

BackgroundThe introduction of multiparameter MRI and novel biomarkers has greatly improved the prediction of clinically significant prostate cancer (csPCa). However, decision-making regarding prostate biopsy and prebiopsy examinations is still difficult. We aimed to establish a quick and economic tool to improve the detection of csPCa based on routinely performed clinical examinations through an automated machine learning platform (AutoML).MethodsThis study included a multicenter retrospective cohort and two prospective cohorts with 4747 cases from 9 hospitals across China. The multimodal data, including demographics, clinical characteristics, laboratory tests, and ultrasound reports, of consecutive participants were retrieved using extract-transform-load tools. AutoML was applied to explore potential data processing patterns and the most suitable algorithm to build the Prostate Cancer Artificial Intelligence Diagnostic System (PCAIDS). The diagnostic performance was determined by the receiver operating characteristic curve (ROC) for discriminating csPCa from insignificant prostate cancer (PCa) and benign disease. The clinical utility was evaluated by decision curve analysis (DCA) and waterfall plots.ResultsThe random forest algorithm was applied in the feature selection, and the AutoML algorithm was applied for model establishment. The area under the curve (AUC) value in identifying csPCa was 0.853 in the training cohort, 0.820 in the validation cohort, 0.807 in the Changhai prospective cohort, and 0.850 in the Zhongda prospective cohort. DCA showed that the PCAIDS was superior to PSA or fPSA/tPSA for diagnosing csPCa with a higher net benefit for all threshold probabilities in all cohorts. Setting a fixed sensitivity of 95%, a total of 32.2%, 17.6%, and 26.3% of unnecessary biopsies could be avoided with less than 5% of csPCa missed in the validation cohort, Changhai and Zhongda prospective cohorts, respectively.ConclusionsThe PCAIDS was an effective tool to inform decision-making regarding the need for prostate biopsy and prebiopsy examinations such as mpMRI. Further prospective and international studies are warranted to validate the findings of this study.Trial registrationChinese Clinical Trial Registry ChiCTR2100048428. Registered on 06 July 2021.