DEEP MULTIMODAL BRAIN NETWORK LEARNING FOR JOINT ANALYSIS OF STRUCTURAL MORPHOMETRY AND FUNCTIONAL CONNECTIVITY.

Previous studies have shown that the left temporoparietal junction (TPJ) plays a critical role in word reading. Nevertheless, there is still controversy surrounding the phonological and semantic functions of the left TPJ. The parietal unified connectivity-biased computation (PUCC) model posits that the function of the left TPJ depends on both the neurocomputation of this local area and its long-range connectivity. To clarify the specific roles of different TPJ subregions in phonological and semantic processing of Chinese characters, the present study used connectivity-based clustering to identify seven subdivisions within the left TPJ, and conducted comprehensive analyses including functional and structural connectivity, univariate and multivariate analyses (i.e., representational similarity analysis, RSA) on multimodal imaging data (task-state fMRI, resting-state fMRI, and diffusion-weighted imaging [DWI]). Functional and structural connectivity analyses revealed that the left anterior TPJ had stronger connections with the phonological network, while the left posterior TPJ had stronger connections with the semantic network. RSA revealed that the left anterior and posterior TPJ represented phonological and semantic information of Chinese characters, respectively. More importantly, the phonological and semantic representations of the left TPJ were respectively correlated with its functional connectivity to the phonological and semantic networks. Altogether, our results provide a more elaborate perspective on the functional dissociation of the left anterior and posterior TPJ in phonological and semantic processing of Chinese characters, and support the PUCC model.

The human entorhinal-hippocampal circuitry is characterized by an information-specific functional organization where two routes, that are preferentially connected to the parahippocampal cortex or the perirhinal and retrosplenial cortices, divide the entorhinal cortex as well as hippocampal subiculum and CA1 subregions.

Subjective cognitive decline (SCD) may be a preclinical stage of Alzheimer's disease (AD). Neuroimaging studies suggest that abnormal brain connectivity plays an important role in the pathophysiology of SCD. However, most previous studies focused on single modalities only. Multimodal combinations can more effectively utilize various information and little is known about their diagnostic value in SCD. One hundred ten SCD individuals and well-matched healthy controls (HCs) were recruited in this study (the primary sample: 35 SCD and 36 HC; the validation sample: 21 SCD and 18 HC). Multimodal imaging data were used to construct functional, anatomical, and morphological networks, respectively. These networks were used in combination with a multiple kernel learning-support vector machine to predict SCD individuals. We validated our model on another independent sample. Multiple linear regression (MLR) analyses were conducted to investigate the relationships among network metrics, cognition, and pathological biomarkers. We found that the characteristics identified from the multimodal network were primarily located in the default mode network (DMN) and salience network (SN), achieving an accuracy of 88.73% (an accuracy of 79.49% for an independent sample) based on the integration of the three modalities. MLR analyses showed that increased AV45 SUVRs were significantly associated with impaired memory function, the enhanced functional connectivity, and the decreased morphological connectivity. This study suggests that abnormal multimodal connections within DMN and SN can be used as effective biomarkers to identify SCD and provide insight into understanding the pathophysiological mechanisms underlying SCD. • Multimodal brain networks improve the detection accuracy of SCD. • Abnormal connections within DMN and SN can be used as effective biomarkers for the identification of SCD.

Brain network analysis is one of the most effective methods for brain disease diagnosis. Existing studies have shown that exploring information from multimodal data is a valuable way to improve the effectiveness of brain network analysis. In recent years, deep learning has received more and more attention due to its powerful feature learning capabilities, and it is natural to introduce this tool into multi-modal brain networks analysis. However, it would face two challenges. One is that brain network is in non-Euclidean domain, so the convolution kernel in deep learning cannot be directly used to brain networks. The other is that most of existing multi-modal brain network analysis methods cannot extract full use of complementary information from distinct modalities. In this paper, we propose a multi-modal non-Euclidean brain network analysis method based on community detection and convolutional autoencoder, which can solve the above two problems simultaneously in one framework (M2CDCA). First, we construct the functional and structural brain network, respectively. Second, we design a multi-modal interactive community detection method that exploits structural modality to guide functional modality for detecting community structure, then readjusts the nodes distribution so that the adjusted brain network can preserve the potential community information and is more suitable for convolution kernels. Finally, we design a dual-channel autoencoder model with self-attention mechanism to capture hierarchical and highly non-linear features, then comprehensively use two modalities information for classification. We evaluate our method on an epilepsy dataset, the experimental results show that our method outperform several state-of-the-art methods.

Unprecedented 4D spatiotemporal infant regional cerebral blood flow framework and region-specific physiology–function coupling across infancy were elucidated, highlighting strong physiology–function coupling specifically at the default-mode network to meet extraneuronal metabolic demand for network emergence.

In this paper, we propose a multiple line extraction method from multimodal data points in high dimensional space. It can sparsely represent multimodal sensor network data by utilizing high correlation among channels in the data. We exploit the idea of Color Lines, which is a model using high correlation among RGB channels in computer vision. It represents real color images as a collection of multiple lines in RGB color space. By extracting color lines from multimodal data, our proposed method can utilize hidden inter-channel relationships unlike conventional methods. We apply the proposed method for compressing a multimodal data matrix and show its effectiveness.

It is an essential task to accurately diagnose cancer subtypes in computational pathology for personalized cancer treatment. Recent studies have indicated that the combination of multimodal data, such as whole slide images (WSIs) and multi-omics data, could achieve more accurate diagnosis. However, robust cancer diagnosis remains challenging due to the heterogeneity among multimodal data, as well as the performance degradation caused by insufficient multimodal patient data. In this work, we propose a novel multimodal co-attention fusion network (MCFN) with online data augmentation (ODA) for cancer subtype classification. Specifically, a multimodal mutual-guided co-attention (MMC) module is proposed to effectively perform dense multimodal interactions. It enables multimodal data to mutually guide and calibrate each other during the integration process to alleviate inter- and intra-modal heterogeneities. Subsequently, a self-normalizing network (SNN)-Mixer is developed to allow information communication among different omics data and alleviate the high-dimensional small-sample size problem in multi-omics data. Most importantly, to compensate for insufficient multimodal samples for model training, we propose an ODA module in MCFN. The ODA module leverages the multimodal knowledge to guide the data augmentations of WSIs and maximize the data diversity during model training. Extensive experiments are conducted on the public TCGA dataset. The experimental results demonstrate that the proposed MCFN outperforms all the compared algorithms, suggesting its effectiveness.

In recent Datascience and AI studies, multimodality has become a key principle for achieving more accurate and dependable results. Managing multimodal data requires specialized platforms and structures to handle diverse data types from single subjects. The established BIDS (Brain Imaging Data Structure) standard faces limitations in supporting multimodal data, as each dataset is assigned to a single study with unique subject identities, preventing integration of multimodal data from the same individual across multiple studies. To address this limitation, this paper introduces FAIR m-BIDS (FAIR Multimodal Brain Imaging Data Structure), extending conventional BIDS by shifting granularity from dataset level to individual data entities. Each brain data file receives an independent GUId-Key (Global Unique Identifier Key), enabling researchers to select and integrate data items from different modalities and studies into customized multimodal datasets. The proposed structure enhances FAIR principles through improved findability, accessibility, interoperability, and reusability. Global identifiers enable tracking anonymized subject data across multiple datasets and modalities, while maintaining compatibility with conventional BIDS standards for advanced AI and neuroscience research applications.

Biotypes of deeply phenotyped depressed patients reflect signatures of adverse childhood experience and depressive cognitive biases.

The clinical progression of Alzheimer's disease( AD ) can't be accurately evaluated by single modality data alone. Multi-modal data have a good effect on the diagnosis of AD. Clarifying the complementarity between modalities is crucial for the assessment of each stage of AD. Few studies have specifically explored the complementarity between different modalities due to the lack of completely aligned and paired multi-modal data and the limitation of sample size. However, collecting the full set of aligned and paired data is expensive or even impractical. In addition, the limited number of samples poses a great challenge to the robustness of the model. In this paper, different machine learning( ML ) methods were used to explore data complementarity between T1-weighted magnetic resonance imaging ( MRI ), cerebrospinal fluid ( CSF ), and fluorodeoxyglucose-positron emission tomography ( FDG-PET ) modalities. The different modal data of Alzheimer's Neuroimaging Initiative ( ADNI ) and the self-extracted neuroimaging data were experimentally explored. Experiments show that there is obvious complementarity between MRI and CSF. By fusing MRI and CSF data, three binary classification tasks using multi-modal fusion data have achieved varying degrees of improvement. At the same time, we also explored the important features of multi-modal fusion data through SHapley Additive exPlanations ( SHAP ), and found that most important features are supported by relevant literature.

BackgroundRecently, it was realized that the functional connectivity networks estimated from actual brain-imaging technologies (MEG, fMRI and EEG) can be analyzed by means of the graph theory, that is a mathematical representation of a network, which is essentially reduced to nodes and connections between them.MethodsWe used high-resolution EEG technology to enhance the poor spatial information of the EEG activity on the scalp and it gives a measure of the electrical activity on the cortical surface. Afterwards, we used the Directed Transfer Function (DTF) that is a multivariate spectral measure for the estimation of the directional influences between any given pair of channels in a multivariate dataset. Finally, a graph theoretical approach was used to model the brain networks as graphs. These methods were used to analyze the structure of cortical connectivity during the attempt to move a paralyzed limb in a group (N=5) of spinal cord injured patients and during the movement execution in a group (N=5) of healthy subjects.ResultsAnalysis performed on the cortical networks estimated from the group of normal and SCI patients revealed that both groups present few nodes with a high out-degree value (i.e. outgoing links). This property is valid in the networks estimated for all the frequency bands investigated. In particular, cingulate motor areas (CMAs) ROIs act as ‘‘hubs’’ for the outflow of information in both groups, SCI and healthy. Results also suggest that spinal cord injuries affect the functional architecture of the cortical network sub-serving the volition of motor acts mainly in its local feature property.In particular, a higher local efficiency El can be observed in the SCI patients for three frequency bands, theta (3-6 Hz), alpha (7-12 Hz) and beta (13-29 Hz).By taking into account all the possible pathways between different ROI couples, we were able to separate clearly the network properties of the SCI group from the CTRL group. In particular, we report a sort of compensatory mechanism in the SCI patients for the Theta (3-6 Hz) frequency band, indicating a higher level of “activation” Ω within the cortical network during the motor task. The activation index is directly related to diffusion, a type of dynamics that underlies several biological systems including possible spreading of neuronal activation across several cortical regions.ConclusionsThe present study aims at demonstrating the possible applications of graph theoretical approaches in the analyses of brain functional connectivity from EEG signals. In particular, the methodological aspects of the i) cortical activity from scalp EEG signals, ii) functional connectivity estimations iii) graph theoretical indexes are emphasized in the present paper to show their impact in a real application.

Recent advances in data collecting methods have enabled life scientists to collect multimodal data from a variety of biological application fields. Multimodal deep learning is becoming more widespread as the capacity of multimodal data streams and deep learning algorithms grows. This leads to the creation of models capable of consistently processing and interpreting multimodal data. The fusion of multimodal data, such as genetic data and histopathological images, is important to better understand cancer heterogeneity and complexity for specific therapies, as well as improving predictions for cancer research. In this brief overview, we present several pioneering deep learning models where we focus on different architectures that has seen use in the medical field, shedding some light on the importance of deep learning in cancer diagnosis considering its immense results in recent years. We particularly set our eyes on multimodal deep learning approaches fusing histopathological images and genomic data to achieve better results than a single modality.

Story listening in children relies on brain regions supporting speech perception, auditory word recognition, syntax, semantics, and discourse abilities, along with the ability to attend and process information (part of executive functions). Speed-of-processing is an early-developed executive function. We used functional and structural magnetic resonance imaging (MRI) to demonstrate the relationship between story listening and speed-of-processing in preschool-age children. Eighteen participants performed story-listening tasks during MRI scans. Functional and structural connectivity analysis was performed using the speed-of-processing scores as regressors. Activation in the superior frontal gyrus during story listening positively correlated with speed-of-processing scores. This region was functionally connected with the superior temporal gyrus, insula, and hippocampus. Fractional anisotropy in the inferior frontooccipital fasciculus, which connects the superior frontal and temporal gyri, was positively correlated with speed-of-processing scores. Our results suggest that speed-of-processing skills in preschool-age children are reflected in functional activation and connectivity during story listening and may act as a biomarker for future academic abilities.

In diagnosing challenging conditions such as Alzheimer's disease (AD), imaging is an important reference. Non-imaging patient data such as patient information, genetic data, medication information, cognitive and memory tests also play a very important role in diagnosis. Effect. However, limited by the ability of artificial intelligence models to mine such information, most of the existing models only use multi-modal image data, and cannot make full use of non-image data. We use a currently very popular pre-trained large language model (LLM) to enhance the model's ability to utilize non-image data, and achieved SOTA results on the ADNI dataset.

Functional magnetic resonance imaging (fMRI) studies with adults provide evidence that functional brain networks, including the default mode network and frontoparietal network, underlie executive functioning (EF). However, given the challenges of using fMRI with infants and young children, little work has assessed the developmental trajectories of these networks or their associations with EF at key developmental stages. More recently, functional near-infrared spectroscopy (fNIRS) has emerged as a promising neuroimaging tool which can provide information on cortical functional networks and can be more easily implemented with young children. Children (N = 207; n = 116 male; n = 167 White) had fNIRS data recorded at infancy, 3, 5, and 7 years of age while watching a 2-min nonsocial video. At 3, 5, and 7 years, children completed behavioral assessments and parents completed questionnaires to assess child EF abilities. Results showed that, although early functional brain network connectivity was not associated with later functional brain connectivity, EF was concurrently and longitudinally associated with functional connectivity levels in both networks. Overall, these results inform the understanding of early emerging neural underpinnings of regulatory abilities and point to considerable change in the composition of functional brain networks and a conservation of function across development.