Multimodal Visual Data Registration for Web-Based Visualization in Media Production

In general, the ‘small sample (n)-large feature ( Open image in new window )’ problem of bioinformatics, image analysis, high throughput molecular screening, astronomy, and other high dimensional applications makes the features highly collinear. In this context, the paper presents a new feature extraction algorithm to address this ‘large Open image in new window small n’ issue associated with multimodal data sets. The proposed algorithm judiciously integrates the concept of both regularization and shrinkage with canonical correlation analysis to extract important features. To deal with the singularity problem, the proposed method increases the diagonal elements of covariance matrices by using regularization parameters, while the off-diagonal elements are decreased by shrinkage coefficients. The concept of hypercuboid equivalence partition matrix of rough hypercuboid approach is used to compute both significance and relevance measures of a feature. The importance of the proposed algorithm over other existing methods is established extensively on real life multimodal omics data set.

PurposeDespite the extensive academic interest in social media sentiment for financial fields, multimodal data in the stock market has been neglected. The purpose of this paper is to explore the influence of multimodal social media data on stock performance, and investigate the underlying mechanism of two forms of social media data, i.e. text and pictures.Design/methodology/approachThis research employs panel vector autoregressive models to quantify the effect of the sentiment derived from two modalities in social media, i.e. text information and picture information. Through the models, the authors examine the short-term and long-term associations between social media sentiment and stock performance, measured by three metrics. Specifically, the authors design an enhanced sentiment analysis method, integrating random walk and word embeddings through Global Vectors for Word Representation (GloVe), to construct a domain-specific lexicon and apply it to textual sentiment analysis. Secondly, the authors exploit a deep learning framework based on convolutional neural networks to analyze the sentiment in picture data.FindingsThe empirical results derived from vector autoregressive models reveal that both measures of the sentiment extracted from textual information and pictorial information in social media are significant leading indicators of stock performance. Moreover, pictorial information and textual information have similar relationships with stock performance.Originality/valueTo the best of the authors’ knowledge, this is the first study that incorporates multimodal social media data for sentiment analysis, which is valuable in understanding pictures of social media data. The study offers significant implications for researchers and practitioners. This research informs researchers on the attention of multimodal social media data. The study’s findings provide some managerial recommendations, e.g. watching not only words but also pictures in social media.

The development of activity recognition based on multi-modal data makes it possible to reduce human intervention in the process of monitoring. This paper proposes an efficient and cost-effective multi-modal sensing framework for activity monitoring, it can automatically identify human activities based on multi-modal data, and provide help to patients with moderate disabilities. The multi-modal sensing framework for activity monitoring relies on parallel processing of videos and inertial data. A new supervised adaptive multi-modal fusion method (AMFM) is used to process multi-modal human activity data. Spatio-temporal graph convolution network with adaptive loss function (ALSTGCN) is proposed to extract skeleton sequence features, and long short-term memory fully convolutional network (LSTM-FCN) module with adaptive loss function is adapted to extract inertial data features. An adaptive learning method is proposed at the decision level to learn the contribution of the two modalities to the classification results. The effectiveness of the algorithm is demonstrated on two public multi-modal datasets (UTD-MHAD and C-MHAD) and a new multi-modal dataset H-MHAD collected from our laboratory. The results show that the performance of the AMFM approach on three datasets is better than the performance of the video or the inertial-based single-modality model. The class-balanced cross-entropy loss function further improves the model performance based on the H-MHAD dataset. The accuracy of action recognition is 91.18%, and the recall rate of falling activity is 100%. The results illustrate that using multiple heterogeneous sensors to realize automatic process monitoring is a feasible alternative to the manual response.

Anger impairs a driver’s control and risk assessment abilities, heightening traffic accident risks. Constructing a multimodal dataset during driving tasks is crucial for accurate anger recognition. This study developed a multimodal physiological -vehicle driving dataset (DPV-MFD) based on drivers’ self-reported anger during simulated driving tasks. In Experiment 1, responses from 624 participants to anger-inducing videos and driving scenarios were collected via questionnaires to select appropriate materials. In Experiments 2 and 3, multimodal dynamic data and self-reported SAM emotion ratings were collected during simulated and real-vehicle tasks, capturing physiological and vehicle responses in neutral and anger states. Spearman’s correlation coefficient analysis validated the DPV-MFD’s effectiveness and explored the relationships between multimodal data and emotional dimensions. The CNN-LSTM deep learning network was used to assess the emotion recognition performance of the DPV-MFD across different time windows, and its applicability in real-world driving scenarios was validated. Compared to using EEG data alone, integrating multimodal data significantly improved anger recognition accuracy, with accuracy and F1 scores rising by 4.49% and 9.14%, respectively. Additionally, real-vehicle data closely matched simulated data, confirming the dataset’s effectiveness for real-world applications. This research is pivotal for advancing emotion-aware human–machine- interaction and intelligent transportation systems.

A rich source of information relevant to smart cities and health care comes from sensors embedded in the environment, or ambient sensors. Strategically responding to sensed data helps healthcare would be smarter. By gaining real time access to this information, city services can respond promptly to urgent health needs and make decisions to avoid unhealthy situations. The emergence of virtual reality and ICT assisted urban planning effectively overcomes these complexities through visualization of big data. Web-based visualization technologies such as virtual reality (VR) present a highly detailed dynamic virtual environment. Also, there is continuously increasing demand for the effective organization of this diversified data, linkage with other data, and the visualization of data. In recent years, there has been a rapid increase in the development of health information systems motivated by legislation intended to protect patients’ information and privacy and government interest in reducing the cost and improving the quality of healthcare. In this paper, we investigate types of physiological content and the related sensors that can be included in a smart city healthcare system. We present a technical healthcare information model with XML in a standardized format for the implementation of a 3D virtual smart city as follows: Firstly, 3D virtual environment and healthcare information systems were confirmed to be dictated by sensor information parameters via XML. Secondly, real-time location, direction, and health data representation for 3D health management sensor data monitoring and real-time monitoring were conducted in XML. The identified key use case physiological sensor-based XML is helpful as it provides a standardized format for the implementation of a 3D virtual smart city. For implementation, X3D was used for visualizing an example health information system.

Traditional public opinion analysis methods mostly focus on single modal data, and it is difficult to make full use of the correlation and complementarity between multi-modal data. In this paper, a multi-modal news data mining and intelligent public opinion analysis model (MM-OpinionNet) is proposed. By fusing multi-modal data, a more comprehensive and accurate public opinion analysis model is constructed. In the research, text data is preprocessed by word segmentation and stop words removal, image and video data are processed by standardization and object detection, and audio data is processed by speech-to-text and acoustic feature extraction. In the feature extraction stage, BERT embedding and TF-IDF are used for text, ResNet-50 for image, 3D-CNN for video and MFCC and prosodic features for audio. The model realizes feature fusion through cross-modal attention mechanism and graph neural network, and introduces cross-modal comparative learning loss to solve the problem of data heterogeneity. The experiment was based on a synthetic multimodal news dataset containing 100,000 news samples encompassing text, images, videos, and audio. The results demonstrated that MM-OpinionNet outperformed unimodal baseline methods and traditional multimodal approaches in event detection, sentiment analysis, and public opinion prediction. It achieved an accuracy of 89.7%, reduced the Root Mean Square Error (RMSE) for sentiment analysis by 34%, and attained a Mean Absolute Percentage Error (MAPE) of 8.3% and a Mean Squared Error (MSE) of 4.1×10³ for public opinion prediction. Ablation studies further confirmed the importance of the cross-modal attention mechanism and multitask learning in enhancing model performance.

In recent Datascience and AI studies, multimodality has become a key principle for achieving more accurate and dependable results. Managing multimodal data requires specialized platforms and structures to handle diverse data types from single subjects. The established BIDS (Brain Imaging Data Structure) standard faces limitations in supporting multimodal data, as each dataset is assigned to a single study with unique subject identities, preventing integration of multimodal data from the same individual across multiple studies. To address this limitation, this paper introduces FAIR m-BIDS (FAIR Multimodal Brain Imaging Data Structure), extending conventional BIDS by shifting granularity from dataset level to individual data entities. Each brain data file receives an independent GUId-Key (Global Unique Identifier Key), enabling researchers to select and integrate data items from different modalities and studies into customized multimodal datasets. The proposed structure enhances FAIR principles through improved findability, accessibility, interoperability, and reusability. Global identifiers enable tracking anonymized subject data across multiple datasets and modalities, while maintaining compatibility with conventional BIDS standards for advanced AI and neuroscience research applications.

Multimedia systems, such as social media platforms, play a crucial role in disseminating vital information during calamities. This information is shared in various formats such as images, text, videos, audio, etc. Therefore, it becomes important to have a system that can identify multimodal data to classify relevant information. This paper proposes a new age classification method for multimodal data using advanced and improved transformer models, such as Vision Transformer and Generative Pre-trained Transformer 2, for image and text classification, respectively. These models were combined using an ensemble model (Random Forest Classifier), achieving an accuracy of 84.66% on the multimodal data. Furthermore, the proposed model demonstrates higher prediction accuracy compared to traditional Convolutional Neural Network (CNN) models which have an accuracy of 71.43%, exceeding it by 13.23%. A comparison with convolutional models is conducted to underscore the advantages of transformer models and to substantiate the necessity of the experiment. Our proposed classification model using Vision Transformer and GPT-2, along with an ensemble model, can be replicated by researchers in disaster management, humanitarian aid organizations, and social media platforms looking to filter and prioritize information during emergencies.

Language understanding is multimodal. During human communication, messages are conveyed not only by words in textual form, but also through speech patterns, gestures or facial emotions of the speakers. Therefore, it is crucial to fuse information from different modalities to achieve a joint comprehension. With the rapid progress in the deep learning field, neural networks have emerged as the most popular approach for addressing multimodal data fusion [1, 6, 7, 12]. While these models can effectively combine multimodal features by learning from data, they nevertheless lack an explicit exhibition of how different modalities are related to each other, due to the inherent low interpretability of neural networks [2]. In the meantime, Quantum Theory (QT) has given rise to principled approaches for incorporating interactions between textual features into a holistic textual representation [3, 5, 8, 10], where the concepts of superposition andentanglement have been universally exploited to formulate interactions. The advantages of those models in capturing complicated correlations between textual features have been observed. We hereby propose the research on quantum-inspired multimodal data fusion, claiming that the limitation of multimodal data fusion can be tackled by quantum-driven models. In particular, we propose to employ superposition to formulate intra-modal interactions while the interplay between different modalities is expected to be captured by entanglement measures. By doing so, the interactions within multimodal data may be rendered explicitly in a unified quantum formalism, increasing the performance and interpretability for concrete multimodal tasks. It will also expand the application domains of quantum theory to multimodal tasks where only preliminary efforts have been made [11]. We therefore aim at answering the following research question: RQ. Can we fuse multimodal data with quantum-inspired models? To answer this question, we propose to fuse multimodal data with complex-valued neural networks, motivated by the theoretical link between neural networks and quantum theory [4] and advances in complex-valued neural networks [9]. Our model begins with a separate complex-valued embedding learned for each unimodal data based on the existing works [5, 10] which inherently assumes superposition between intra-modal features. Then we construct a many-body system in entangled state for multimodal data, where cross-modality interactions are naturally reflected by entanglement measures. Quantum measurement operators are applied to the entanglement state to address a concrete multimodal task at hand. The whole process is instrumented by a complex-valued neural network, which is able to learn how multimodal features are combined from data, and at the same time explain the combination by means of quantum superposition and entanglement measures. We plan to examine our proposed models on CMU-MOSI [12] and CMU-MOSEI [1] which are benchmarking multimodal sentiment analysis datasets. The dataset targets at classifying sentiment into 2, 5 or 7 classes with the input of textual, visual and acoustic features. We expect to see comparable effectiveness to state-of-the-art models, and we will explore superposition and entanglement measures to better understand the inter-modal interactions.

ObjectiveEvaluating the performance of multiple complex models, such as those found in biology, medicine, climatology, and machine learning, using conventional approaches is often challenging when using various evaluation metrics simultaneously. The traditional approach, which relies on presenting multi-model evaluation scores in the table, presents an obstacle when determining the similarities between the models and the order of performance. MethodsBy combining statistics, information theory, and data visualization, juxtaposed Taylor and Mutual Information Diagrams permit users to track and summarize the performance of one model or a collection of different models. To uncover linear and nonlinear relationships between models, users may visualize one or both charts. ResultsOur library presents the first publicly available implementation of the Mutual Information Diagram and its new interactive capabilities, as well as the first publicly available implementation of an interactive Taylor Diagram. Extensions have been implemented so that both diagrams can display temporality, multimodality, and multivariate data sets, and feature one scalar model property such as uncertainty. Our library, named polar-diagrams, supports both continuous and categorical attributes. ConclusionThe library can be used to quickly and easily assess the performances of complex models, such as those found in machine learning, climate, or biomedical domains.

A kernel machine method for detecting higher order interactions in multimodal datasets: Application to schizophrenia

Table grape production and quality is highly influenced by environmental conditions, with abrupt weather changes increasing the risk of vineyard diseases. The integration of smart agriculture technologies enables early detection and effective management of these threats, improving crop quality and yield. This work proposes the GraDA system, a data collection and grape disease risk assessment system, consisting of three main components, namely the GraDA platform, a mobile application and an AI-driven grape disease risk prediction model. The mobile application is responsible for collecting images of grapes and leaves, while the AI-driven disease risk prediction model processes multi-modal input data—grape images, leaf images, and meteorological data—to achieve enhanced grape disease risk prediction accuracy. Finally, the GraDA platform is concerned with the collection and visualization of imaging and meteorological data through a web-based interface, enabling agronomists and farmers to monitor in real-time the weather conditions and the estimated disease risk scores, as well as manage farming activities, thus facilitating data-driven decision-making for vineyard health optimization. The proposed AI model’s accuracy was evaluated in a multi-modal dataset, achieving a performance above 93% in grape disease risk estimation, while the usability of the overall GraDA system was assessed through questionnaires distributed to agronomists and farmers, leading to a score of 79.83% in the SUS scale, indicating excellent usability and user satisfaction from the proposed system. Finally, a pilot study of the GraDA system was performed to assess the applicability of the system in real-life scenarios.

The emergence of visual big data is a doubleedged sword to mobile communications. On one hand, the massive scale of visual data transmission brings a huge challenge to the RAN. On the other, the abundant information in visual big data may improve the spectrum efficiency and robustness of visual communications. In this article, we propose a DaC-RAN architecture for visual communications, which integrates the ideas of SDN and C-RAN. We propose to separate the control and data planes in the conventional infrastructure, and integrate a new data plane specifically designed for visual communications into the virtual base station. We demonstrate the proposed DaC-RAN architecture through a practical visual communication system based on pseudo-analog transmission and CSbased data sampling and reconstruction. The correlation information retrieved from visual big data is utilized as prior knowledge in CS decoding. Preliminary simulation evaluations show that significant gain is achieved in spectrum efficiency over the conventional method.

In smart homes, data generated from real-time sensors for human activity recognition is complex, noisy and imbalanced. It is a significant challenge to create machine learning models that can classify activities which are not as commonly occurring as other activities. Machine learning models designed to classify imbalanced data are biased towards learning the more commonly occurring classes. Such learning bias occurs naturally, since the models better learn classes which contain more records. This paper examines whether fusing real-world imbalanced multi-modal sensor data improves classification results as opposed to using unimodal data; and compares deep learning approaches to dealing with imbalanced multi-modal sensor data when using various resampling methods and deep learning models. Experiments were carried out using a large multi-modal sensor dataset generated from the Sensor Platform for HEalthcare in a Residential Environment (SPHERE). The data comprises 16104 samples, where each sample comprises 5608 features and belongs to one of 20 activities (classes). Experimental results using SPHERE demonstrate the challenges of dealing with imbalanced multi-modal data and highlight the importance of having a suitable number of samples within each class for sufficiently training and testing deep learning models. Furthermore, the results revealed that when fusing the data and using the Synthetic Minority Oversampling Technique (SMOTE) to correct class imbalance, CNN-LSTM achieved the highest classification accuracy of 93.67% followed by CNN, 93.55%, and LSTM, i.e. 92.98%.

BackgroundThree-dimensional (3-D) visualization of multimodality neuroimaging data provides a powerful technique for viewing the relationship between structure and function. A number of applications are available that include some aspect of 3-D visualization, including both free and commercial products. These applications range from highly specific programs for a single modality, to general purpose toolkits that include many image processing functions in addition to visualization. However, few if any of these combine both stand-alone and remote multi-modality visualization in an open source, portable and extensible tool that is easy to install and use, yet can be included as a component of a larger information system.ResultsWe have developed a new open source multimodality 3-D visualization application, called MindSeer, that has these features: integrated and interactive 3-D volume and surface visualization, Java and Java3D for true cross-platform portability, one-click installation and startup, integrated data management to help organize large studies, extensibility through plugins, transparent remote visualization, and the ability to be integrated into larger information management systems. We describe the design and implementation of the system, as well as several case studies that demonstrate its utility. These case studies are available as tutorials or demos on the associated website: .ConclusionMindSeer provides a powerful visualization tool for multimodality neuroimaging data. Its architecture and unique features also allow it to be extended into other visualization domains within biomedicine.