MMKT: Multimodal Sentiment Analysis Model Based on Knowledge-Enhanced and Text-Guided Learning

Multimodal sentiment analysis with unidirectional modality translation

Multimodal sentiment analysis (MMSA) is a research method that extracts effective information from heterogeneous modal information. Then, MMSA processes the multimodal data and performs sentiment analysis. Along with big data and machine learning development, multimodal sentiment analysis has become a hot research direction in multimodal learning and natural language processing. Although various feature extraction methods and information fusion methods have been continuously proposed, challenges exist in MMSA research. First, in terms of feature extraction, pre-trained models trained with many data sets can obtain higher quality features, but research on how to use these feature extraction methods to extract the best features is still needed. Currently, the more popular feature fusion methods do not focus on the interaction between multiple modal information and the retention of basic information. To overcome these problems a multimodal sentiment analysis model utilizes text features as core modal features, using video and audio modal features as auxiliary modal features, multimodal feature modality attention mechanism to extract the intrinsic connection between different modalities. The attention mechanism uses the features of video modality and audio modality as the focus and then enhances the text modality with the fusion of video modality and modality. To improve the quality of extracted features, this method chooses the transfer learning training method and uses the pre-trained model for processing. This research uses the CMU-MOSI dataset to test the proposed method. Experimental results show that the performance of the proposed model in emotion score prediction and emotion classification tasks exceeds traditional methods and baseline methods.

The homogeneity and heterogeneity across modalities are critical factors that influence multimodal fusion. In Multimodal Sentiment Analysis (MSA), the inherent textual information within the audio modality induces cross-modality homogeneity with the text modality. Conversely, the mutual independence between text and vision modalities results in their cross-modal heterogeneity. Although existing disentangle-based methods achieve notable performance gains by separating modality features into distinct subspaces, they overlook the characteristics of cross-modality heterogeneity and homogeneity among different modalities. To this end, we propose a novel Modality-aware Disentangle and Fusion (MDF) framework to investigate the role of core modality features. Specifically, we first use text as the anchor to disentangle the audio modality and extract its unique modality-specific features, thereby establishing cross-modal heterogeneity among text, audio, and vision. We then introduce a Cross-Modality Heterogeneity Enhancement (CHE) module to refine these features, further reinforcing their heterogeneous characteristics. Finally, a Modality Adaptive Weighting (MAW) module is employed to dynamically assign weights to the text, sound, and vision modalities based on their potential contributions to sentiment prediction, achieving a more effective multimodal representation for MSA. Experimental evaluations on different benchmarks demonstrate MDF's superiority, with extensive ablation studies confirming its effectiveness.

Multimodal Aspect-Based Sentiment Analysis (MABSA) is an essential task in sentiment analysis that has garnered considerable attention in recent years. Typical approaches in MABSA often utilize cross-modal Transformers to capture interactions between textual and visual modalities. However, bridging the semantic gap between modalities spaces and addressing interference from irrelevant visual objects at different scales remains challenging. To tackle these limitations, we present the Multi-level Textual-Visual Alignment and Fusion Network (MTVAF) in this work, which incorporates three auxiliary tasks. Specifically, MTVAF first transforms multi-level image information into image descriptions, facial descriptions, and optical characters. These are then concatenated with the textual input to form a textual+visual input, facilitating comprehensive alignment between visual and textual modalities. Next, both inputs are fed into an integrated text model that incorporates relevant visual representations. Dynamic attention mechanisms are employed to generate visual prompts to control cross-modal fusion. Finally, we align the probability distributions of the textual input space and the textual+visual input space, effectively reducing noise introduced during the alignment process. Experimental results on two MABSA benchmark datasets demonstrate the effectiveness of the proposed MTVAF, showcasing its superior performance compared to state-of-the-art approaches. Our codes are available at https://github.com/MKMaS-GUET/MTVAF.

With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words representations and average facial expression intensity, we propose a novel deep architecture for multimodal sentiment analysis that is able to perform modality fusion at the word level. In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules. The Gated Multimodal Embedding allows us to alleviate the difficulties of fusion when there are noisy modalities. The LSTM with Temporal Attention can perform word level fusion at a finer fusion resolution between the input modalities and attends to the most important time steps. As a result, the GME-LSTM(A) is able to better model the multimodal structure of speech through time and perform better sentiment comprehension. We demonstrate the effectiveness of this approach on the publicly-available Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset by achieving state-of-the-art sentiment classification and regression results. Qualitative analysis on our model emphasizes the importance of the Temporal Attention Layer in sentiment prediction because the additional acoustic and visual modalities are noisy. We also demonstrate the effectiveness of the Gated Multimodal Embedding in selectively filtering these noisy modalities out. These results and analysis open new areas in the study of sentiment analysis in human communication and provide new models for multimodal fusion.

Multimodal sentiment analysis is an extended approach to traditional language-based sentiment analysis, which uses other relevant modality data. Multimodal sentiment analysis usually applies visual, textual, and acoustic representations for sentiment prediction. Recently, various data fusion methodologies have been proposed for multimodal sentiment analysis. In most cases, textual modality plays a major role, and visual and acoustic modalities are used as auxiliary sources for multimodal sentiment analysis. However, in general multimedia such as video, text transcripts of an individual's speech are not provided. Research on an audio-visual sentiment analysis methodology that does not depend on text modality is essential for multimodal sentiment analysis in real-world industrial applications. Therefore, it is important to improve audio-visual sentiment analysis because it currently exhibits lower performance than multimodal sentiment analysis, including text modality. In this paper, we propose heterogeneous modality transfer learning (HMTL) to utilize the knowledge of aligned text data as a source modality in transfer learning to improve audio-visual sentiment analysis performance. Our approach uses a decoder and adversarial learning techniques to reduce the gap between the source and target modalities in the embedded space for multimodal representation. Our proposed methodology experimentally outperformed recent unimodal and bimodal audio-visual sentiment analysis achievements.

Integrating color cues to improve multimodal sentiment analysis in social media

The goal of Visual Answer Localization (VAL) is to locate a video segment that answers a given question. Existing methods either focus solely on visual modality or integrate visual and subtitle modalities. However, these methods neglect the audio modality in videos, consequently leading to incomplete input information and poor consistency in the VAL task. In this paper, we propose a unified Audio-Visual-Textual Span Localization (AVTSL) method that incorporates audio modality to augment both visual and textual representations for the VAL task. Specifically, we design a network structure with three interacting predictors, each corresponding to the audio, visual, and text modalities. Each predictor generates predictions based on its respective modality. To maintain consistency across the predicted results, we design an Audio-Visual-Textual Consistency module. This module utilizes a Dynamic Triangular Loss (DTL) function, allowing each modality’s predictor to dynamically learn from the others. This collaborative learning ensures that the model generates consistent and comprehensive answers. Extensive experiments show that our proposed method outperforms several state-of-the-art (SOTA) methods, which demonstrates the effectiveness of the audio modality. Code is available at https://github.com/binbin2xs/AVTSL.

TCMT: Target-oriented Cross Modal Transformer for Multimodal Aspect-Based Sentiment Analysis

Aiming at the existing multimodal sentiment analysis approaches, which include inadequate extraction of unimodal features, redundancy of independent modal features, insufficient analysis of semantic correlation between data and insufficient fusion, a Web-Semantic Enhanced Multimodal Sentiment Analysis Using Multilayer Cross-Attention Fusion is proposed. The model utilizes deep learning (including XLNet, ResNeSt, and convolutional neural networks) to extract high-level features from text, audio, and visual modes through self-attention mechanisms, and improves the accuracy of emotion classification through multimodal fusion. The results of experiments demonstrate that the suggested MCFMSA can achieve Acc-2, Acc-3, F1, and MAE values of 89.7%, 85.2%, 89.3%, and 0.466 on the CMU-MOSI dataset, respectively; and on the CMU-MOSEI dataset, Acc-2, Acc-3, F1, and MAE values of 88.7%, 82.5%, 86.5%, and 0.475. All of them are significantly improved compared to several other advanced multimodal sentiment analysis methods, which can enhance the accuracy of sentiment classification.

Meetings are a common activity in professional contexts, and it remains challenging to endow vocal assistants with advanced functionalities to facilitate meeting management. In this context, a task like active speaker detection can provide useful insights to model interaction between meeting participants. Detection of the active speaker can be performed using only video based on the movements of the participants of a meeting. Depending on the assistant design and each participant position regarding the device, active speaker detection can benefit from information coming from visual and audio modalities. Motivated by our application context related to advanced meeting assistant, we want to combine audio and visual information to achieve the best possible performance. In this paper, we propose two different types of fusion (naive fusion and attention-based fusion) for the detection of the active speaker, combining two visual modalities and an audio modality through neural networks. In addition, the audio modality is mainly processed using neural networks. For comparison purpose, classical unsupervised approaches for audio feature extraction are also used. We expect visual data centered on the face of each participant to be very appropriate for detecting voice activity, based on the detection of lip and facial gestures. Thus, our baseline system uses visual data (video) and we chose a 3D Convolutional Neural Network (CNN) architecture, which is effective for simultaneously encoding appearance and movement. To improve this system, we supplemented the visual information by processing the audio stream with a CNN or an unsupervised speaker diarization system. We have further improved this system by adding visual modality information using motion through optical flow. We evaluated our proposal with a public and state-of-the-art benchmark: the AMI corpus. We analysed the contribution of each system to the merger carried out in order to determine if a given participant is currently speaking. We also discussed the results we obtained. Besides, we have shown that, for our application context, adding motion information greatly improves performance. Finally, we have shown that attention-based fusion improves performance while reducing the standard deviation.

Multimodal aspect-based sentiment analysis (MABSA) is an essential task in the field of sentiment analysis, which still confronts several critical challenges. The first challenge is how to effectively capture key information within both image and text features to enhance the recognition and understanding of complex sentiment expressions. The second challenge is how to achieve cross-modal alignment of multi-granularity text features and image features. The third challenge is how to narrow the semantic gap between image modality and text modality through effective cross-modal feature fusion. To address these issues, a framework that leverages external knowledge and multi-granularity image and text features (EKMG) is proposed. Firstly, an external knowledge enhanced semantic extraction module is introduced to fuse external knowledge with image features and text features, thereby capturing the key information from texts and images. Secondly, we design a multi-granularity image-text contrastive learning module. This module initially introduces a graph attention network and a novel cross-modal fusion mechanism to align image features and text features at multiple granularities. Additionally, the module employs an image-text contrastive learning strategy to narrow the semantic gap between different modalities. Experimental results on two public benchmark datasets demonstrate that EKMG achieves significant performance improvements compared to state-of-the-art baseline models.

Multimodal sentiment analysis is an important research area that predicts speaker’s sentiment tendency through features extracted from textual, visual and acoustic modalities. The central challenge is the fusion method of the multimodal information. A variety of fusion methods have been proposed, but few of them adopt end-to-end translation models to mine the subtle correlation between modalities. Enlightened by recent success of Transformer in the area of machine translation, we propose a new fusion method, TransModality, to address the task of multimodal sentiment analysis. We assume that translation between modalities contributes to a better joint representation of speaker’s utterance. With Transformer, the learned features embody the information both from the source modality and the target modality. We validate our model on multiple multimodal datasets: CMU-MOSI, MELD, IEMOCAP. The experiments show that our proposed method achieves the state-of-the-art performance.

VLP2MSA: Expanding vision-language pre-training to multimodal sentiment analysis

Multimodal sentiment analysis (MSA) has recently encountered two major challenges: non‐textual modalities are often affected by noise, and sentiment intensity differences are difficult to capture. To address these issues, we propose a Sentiment Intensity Contrastive Text‐Enhanced Fusion Network (SICTEF Net), which achieves deep collaboration among text, audio, and visual modalities through three key mechanisms. First, a grouped‐channel‐attention based Feature Enhancement Module (EMA) is designed to mitigate modality‐specific noise and emphasize emotion‐sensitive cues by combining spatial–channel interaction mapping with dual‐branch attention fusion. Second, a text‐centered cross‐modal fusion mechanism is introduced, where bidirectional multi‐head self‐attention and a residual‐enhanced encoder jointly enable complementary mappings between text and non‐text modalities, thereby producing intermediate representations that preserve semantic primacy while incorporating fine‐grained complementary information. Third, a sentiment‐intensity weighted contrastive learning strategy dynamically assigns weights to positive and negative sample pairs according to their sentiment intensity differences, allowing the model to more precisely distinguish samples with varying degrees of similarity in the embedding space. Experimental evaluation on the CMU‐MOSI and CMU‐MOSEI datasets demonstrates that SICTEF Net consistently outperforms state‐of‐the‐art baselines in binary accuracy, F1 score, seven‐class accuracy, mean absolute error (MAE), and Pearson correlation. Comprehensive ablation studies further confirm the complementary benefits of EMA, the text‐enhanced Transformer, and sentiment‐intensity contrastive learning. These results indicate that combining text‐driven deep interaction, non‐text modality enhancement via channel attention, and contrastive learning can improve the accuracy and robustness of multimodal sentiment analysis.