Multi-modal brain data exploration software for imaging transcriptomics: A comparative evaluation and future directions

The clinical progression of Alzheimer's disease( AD ) can't be accurately evaluated by single modality data alone. Multi-modal data have a good effect on the diagnosis of AD. Clarifying the complementarity between modalities is crucial for the assessment of each stage of AD. Few studies have specifically explored the complementarity between different modalities due to the lack of completely aligned and paired multi-modal data and the limitation of sample size. However, collecting the full set of aligned and paired data is expensive or even impractical. In addition, the limited number of samples poses a great challenge to the robustness of the model. In this paper, different machine learning( ML ) methods were used to explore data complementarity between T1-weighted magnetic resonance imaging ( MRI ), cerebrospinal fluid ( CSF ), and fluorodeoxyglucose-positron emission tomography ( FDG-PET ) modalities. The different modal data of Alzheimer's Neuroimaging Initiative ( ADNI ) and the self-extracted neuroimaging data were experimentally explored. Experiments show that there is obvious complementarity between MRI and CSF. By fusing MRI and CSF data, three binary classification tasks using multi-modal fusion data have achieved varying degrees of improvement. At the same time, we also explored the important features of multi-modal fusion data through SHapley Additive exPlanations ( SHAP ), and found that most important features are supported by relevant literature.

Safe: A multimodal, scalable and clinically-oriented comprehensive framework for synthetic data validation in hematology

Various machine learning methods have been proposed for predicting progression of patients with mild cognitive impairment (MCI) to Alzheimer's disease (AD) using neuroimaging data. Even though the vast majority of these works use the public dataset ADNI, reproducing their results is complicated because they often do not make available elements that are essential for reproducibility, such as selected participants and input data, image preprocessing and cross-validation procedures. Comparability is also an issue. Specially, the influence of different components like preprocessing, feature extraction or classification algorithms on the performance is difficult to evaluate. Finally, these studies rarely compare their results to models built from clinical data only, a critical aspect to demonstrate the utility of neuroimaging. In our previous work, 1, 2 we presented a framework for reproducible and objective classification experiments in AD, that included automatic conversion of ADNI database into the BIDS community standard, image preprocessing pipelines and machine learning evaluation. We applied this framework to perform unimodal classifications of T1 MRI and FDG-PET images. In the present paper, we extend this work to the combination of multimodal clinical and neuroimaging data. All experiments are based on standard approaches (namely SVM and random forests). In particular, we assess the added value of neuroimaging over using only clinical data. We first demonstrate that using only demographic and clinical data (gender, education level, MMSE, CDR sum of boxes, ADASCog) results in a balanced accuracy of 75% (AUC of 0.84). This performance is higher than that of standard neuroimaging-based classifiers. We then propose a simple trick to improve the performance of neuroimaging-based classifiers: training from AD patients and controls (rather than from MCI patients) improves the performance of FDG-PET classification by 5 percent points, reaching the level of the clinical classifier. Finally, combining clinical and neuroimaging data, prediction results further improved to 80% balanced accuracy and an AUC of 0.88). These prediction accuracies, obtained in a reproducible way, provide a base to develop on top of it and, to compare against, more sophisticated methods. All the code of the framework and the experiments is publicly available at https://github.com/aramis-lab/AD-ML

The Big Data era in Biomedical research has resulted in large-cohort data repositories such as The Cancer Genome Atlas (TCGA). These repositories routinely contain hundreds of matched patient samples for genomic, proteomic, imaging, and clinical data modalities, enabling holistic and multi-modal integrative analysis of human disease. Using TCGA renal and ovarian cancer data, we conducted a novel investigation of multi-modal data integration by combining histopathological image and RNA-seq data. We compared the performances of two integrative prediction methods: majority vote and stacked generalization. Results indicate that integration of multiple data modalities improves prediction of cancer grade and outcome. Specifically, stacked generalization, a method that integrates multiple data modalities to produce a single prediction result, outperforms both single-data-modality prediction and majority vote. Moreover, stacked generalization reveals the contribution of each data modality (and specific features within each data modality) to the final prediction result and may provide biological insights to explain prediction performance.

Imaging transcriptomics: Convergent cellular, transcriptomic, and molecular neuroimaging signatures in the healthy adult human brain.

More and more healthcare data are becoming readily available nowadays. These data can help the healthcare professionals and patient themselves to better understand the patient status and potentially lead to improved care quality. However, the analysis of these data are challenging because they are large-scale and heterogeneous, high-dimensional and sparse, temporal but irregularly sampled. In this paper, we propose a method called Double Core Memory Networks (DCMN) to integrate information from different modalities of the longitudinal patient data and learn a joint patient representation effective for downstream analytical tasks such as risk prediction. DCMN is designed not only to disentangle the temporal and non-linear intra-modal dependencies for the data within each modality but also to capture the long-term inter-modal interactions. DCMN models are the end-to-end memory networks with two external memory cores where each modality of data is compressed and stored. Each memory core has an information-flow controller named query to interact with an external memory module. In addition, we incorporate a gating mechanism into basic DCMN model to perform dynamic regulation of memory interaction. DCMN models have multiple computational layers (hops) allowing data of different modalities interacting with each other recurrently along with a mechanism of alternating access of external memory for each memory core hop-by-hop. We evaluate DCMN models on two outcome prediction tasks, including a mortality prediction on the public Medical Information Mart for Intensive Care III (MIMIC-III) database and a cost prediction on the Hospital Quality Monitoring System (HQMS) dataset. Experimental results demonstrate that our DCMN models are more competitive over the baseline methods in the multimodal prediction setting.

Fusion of multimodal medical data provides multifaceted, disease-relevant information for diagnosis or prognosis prediction modeling. Traditional fusion strategies such as feature concatenation often fail to learn hidden complementary and discriminative manifestations from high-dimensional multimodal data. To this end, we proposed a methodology for the integration of multimodality medical data by matching their moments in a latent space, where the hidden, shared information of multimodal data is gradually learned by optimization with multiple feature collinearity and correlation constrains. We first obtained the multimodal hidden representations by learning mappings between the original domain and shared latent space. Within this shared space, we utilized several relational regularizations, including data attribute preservation, feature collinearity and feature-task correlation, to encourage learning of the underlying associations inherent in multimodal data. The fused multimodal latent features were finally fed to a logistic regression classifier for diagnostic prediction. Extensive evaluations on three independent clinical datasets have demonstrated the effectiveness of the proposed method in fusing multimodal data for medical prediction modeling.

In recent years deep-learning methods have demon-strated impressive results in various domains of computer vision, including medical imaging. This paper examines the possibility of leveraging deep-learning concepts in designing a computer system that could help clinicians make accurate Alzheimer disease (AD) diagnosis by retrieving the most similar archived brain scans of patients with already known diagnoses. We implement a siamese network with ResNet-50 twin subnetworks and train it on the MRI data obtained from ADNI (Alzheimer's Disease Neu-roimaging Initiative) dataset. Four different approaches for slice extraction from MRI volume are considered: using the three slices from the same plane (axial, coronal or sagittal) and combining one slice from each plane. The final performance of the CBIR system on new patient's data based only on MR neuroimaging modality shows limited and comparable performance with all four approaches and leaves space for further enhancements, including complementing neuroimaging MRI data with other data modalities relevant for AD detection.

Given the rapidly decreasing cost and increasing speed and accessibility of massively parallel technologies, the integration of comprehensive genomic, transcriptomic, and proteomic data into a "multi-omics" diagnostic pipeline is within reach. Even though genomic analysis has the capability to reveal all possible perturbations in our genetic code, analysis typically reaches a diagnosis in just 35% of cases, with a diagnostic gap arising due to limitations in prioritization and interpretation of detected variants. Here we review the utility of complementing genetic data with transcriptomic data and give a perspective for the introduction of proteomics into the diagnostic pipeline. Together these methodologies enable comprehensive capture of the functional consequence of variants, unobtainable by the analysis of each methodology in isolation. This facilitates functional annotation and reprioritization of candidate genes and variants-a promising approach to shed light on the underlying molecular cause of a patient's disease, increasing diagnostic rate, and allowing actionability in clinical practice.

MIGNON is a workflow for the analysis of RNA-Seq experiments, which not only efficiently manages the estimation of gene expression levels from raw sequencing reads, but also calls genomic variants present in the transcripts analyzed. Moreover, this is the first workflow that provides a framework for the integration of transcriptomic and genomic data based on a mechanistic model of signaling pathway activities that allows a detailed biological interpretation of the results, including a comprehensive functional profiling of cell activity. MIGNON covers the whole process, from reads to signaling circuit activity estimations, using state-of-the-art tools, it is easy to use and it is deployable in different computational environments, allowing an optimized use of the resources available.

Individuals with Parkinson’s disease present with a complex clinical phenotype, encompassing sleep, motor, cognitive, and affective disturbances. However, characterizations of PD are typically made for the “average” patient, ignoring patient heterogeneity and obscuring important individual differences. Modern large-scale data sharing efforts provide a unique opportunity to precisely investigate individual patient characteristics, but there exists no analytic framework for comprehensively integrating data modalities. Here we apply an unsupervised learning method—similarity network fusion—to objectively integrate MRI morphometry, dopamine active transporter binding, protein assays, and clinical measurements from n = 186 individuals with de novo Parkinson’s disease from the Parkinson’s Progression Markers Initiative. We show that multimodal fusion captures inter-dependencies among data modalities that would otherwise be overlooked by field standard techniques like data concatenation. We then examine how patient subgroups derived from the fused data map onto clinical phenotypes, and how neuroimaging data is critical to this delineation. Finally, we identify a compact set of phenotypic axes that span the patient population, demonstrating that this continuous, low-dimensional projection of individual patients presents a more parsimonious representation of heterogeneity in the sample compared to discrete biotypes. Altogether, these findings showcase the potential of similarity network fusion for combining multimodal data in heterogeneous patient populations.

Ontology mapping as a semantic data integration approach has evolved from traditional data integration solutions. The core problems and open issues related to early data integration approaches are also applicable to ontology mapping on the Semantic Web community. Therefore, in this review the authors present the related literature, starting from the traditional data integration approaches, in order to highlight the evolution of data integration from the early approaches. Once the roots of semantic data integration have been presented, the authors proceed to introduce the state-of-the-art of the ontology mappings systems including the early approaches and the systems that can be compared through the Ontology Alignment Initiative (OAEI).

Ontology mapping as a semantic data integration approach has evolved from traditional data integration solutions. The core problems and open issues related to early data integration approaches are also applicable to ontology mapping on the Semantic Web community. Therefore, in this review the authors present the related literature, starting from the traditional data integration approaches, in order to highlight the evolution of data integration from the early approaches. Once the roots of semantic data integration have been presented, the authors proceed to introduce the state-of-the-art of the ontology mappings systems including the early approaches and the systems that can be compared through the Ontology Alignment Initiative (OAEI).

BackgroundDetermination of batch-to-batch consistency of botanical drugs (BDs) has long been the bottleneck in quality evaluation primarily due to the chemical diversity inherent in BDs. This diversity presents an obstacle to achieving comprehensive standardization for BDs. Basically, a single detection mode likely leads to substandard analysis results as different classes of structures always possess distinct physicochemical properties. Whereas representing a workaround for multi-target standardization using multi-modal data, data processing for information from diverse sources is of great importance for the accuracy of classification.MethodsIn this research, multi-modal data of 78 batches of Guhong injections (GHIs) consisting of 52 normal and 26 abnormal samples were acquired by employing HPLC-UV, -ELSD, and quantitative 1H NMR (q1HNMR), of which data obtained was then individually used for Pearson correlation coefficient (PCC) calculation and partial least square-discriminant analysis (PLS-DA). Then, a mid-level data fusion method with data containing qualitative and quantitative information to establish a support vector machine (SVM) model for evaluating the batch-to-batch consistency of GHIs.ResultsThe resulting outcomes showed that datasets from one detection mode (e.g., data from UV detectors only) are inadequate for accurately assessing the product's quality. The mid-level data fusion strategy for the quality evaluation enabled the classification of normal and abnormal batches of GHIs at 100% accuracy.ConclusionsA quality assessment strategy was successfully developed by leveraging a mid-level data fusion method for the batch-to-batch consistency evaluation of GHIs. This study highlights the promising utility of data from different detection modes for the quality evaluation of BDs. It also reminds manufacturers and researchers about the advantages of involving data fusion to handle multi-modal data. Especially when done jointly, this strategy can significantly increase the accuracy of product classification and serve as a capable tool for studies of other BDs.

In this paper, we leverage multimodal data to classify minerals using a multi-stream neural network. In a previous study on the Tinto dataset, which consisted of a 3D hyperspectral point cloud from the open-pit mine Corta Atalaya in Spain, we successfully identified mineral classes by employing various deep learning models. However, this prior work solely relied on hyperspectral data as input for the deep learning models. In this study, we aim to enhance accuracy by incorporating multimodal data, which includes hyperspectral images, RGB images, and a 3D point cloud. To achieve this, we have adopted a graph-based neural network, known for its efficiency in aggregating local information, based on our past observations where it consistently performed well across different hyperspectral sensors. Subsequently, we constructed a multi-stream neural network tailored to handle multimodality. Additionally, we employed a channel attention module on the hyperspectral stream to fully exploit the spectral information within the hyperspectral data. Through the integration of multimodal data and a multi-stream neural network, we achieved a notable improvement in mineral classification accuracy: 19.2%, 4.4%, and 5.6% on the LWIR, SWIR, and VNIR datasets, respectively.