Multi-modal Representation Learning with Self-adaptive Threshold for Commodity Verification

Multimodal representations play an important role in multimodal learning tasks, including cross-modal retrieval and intra-modal clustering. However, existing multimodal representation learning approaches focus on building one common space by aligning different modalities and ignore the complementary information across the modalities, such as the intra-modal local structures. In other words, they only focus on the object-level alignment and ignore structure-level alignment. To tackle the problem, we propose a novel symmetric multimodal representation learning framework by transferring local structures across different modalities, namely MTLS. A customized soft metric learning strategy and an iterative parameter learning process are designed to symmetrically transfer local structures and enhance the cluster structures in intra-modal representations. The bidirectional retrieval loss based on multi-layer neural networks is utilized to align two modalities. MTLS is instantiated with image and text data and shows its superior performance on image-text retrieval and image clustering. MTLS outperforms the state-of-the-art multimodal learning methods by up to 32% in terms of R@1 on text-image retrieval and 16.4% in terms of AMI onclustering.

Multimodal representation learning has shown promising improvements on various vision-language tasks (e.g., image-text retrieval, visual question answering, etc) and has significantly advanced the development of multimedia information systems. Most existing methods excel at building global-level alignment between vision and language while lacking effective fine-grained image-text interaction. In this paper, we propose a jointly masked multimodal modeling method to learn fine-grained multimodal representations. Our method performs joint masking on image-text input and integrates both implicit and explicit targets for the masked signals to recover. The implicit target provides a unified and debiased objective for vision and language, where the model predicts latent multimodal representations of the unmasked input. The explicit target further enriches the multimodal representations by recovering high-level and semantically meaningful information: momentum visual features of image patches and concepts of word tokens. Through such a masked modeling process, our model not only learns fine-grained multimodal interaction, but also avoids the semantic gap between high-level representations and low-or mid-level prediction targets (e.g., image pixels, discrete vision tokens), thus producing semantically rich multimodal representations that perform well on both zero-shot and fine-tuned settings. Our pre-trained model (named MAMO) achieves state-of-the-art performance on various downstream vision-language tasks, including image-text retrieval, visual question answering, visual reasoning, and weakly-supervised visual grounding.

Multi-modal knowledge graphs representation learning via multi-headed self-attention

Aiming at the problem of 5G multimedia heterogeneous multimodal network representation learning, this paper proposes a collaborative multimodal heterogeneous network representation learning method based on attention mechanism. This method learns different representations for nodes based on heterogeneous network structure information and multimodal content and designs an attention mechanism to learn weights for different representations to fuse them to obtain robust node representations. Combining the general process of exploring the college physical education model and the characteristics of the multimedia network classroom environment, this article constructs the process of exploring the college physical education teaching model of the multimedia network classroom. Through the research and practice of the inquiry college physical education teaching model in the multimedia network classroom, it is verified that the implementation of the inquiry college physical education teaching in the multimedia network classroom can effectively influence and increase the students’ interest in learning and stimulate the students’ inner learning motivation. Through the guidance and training of teachers, a variety of disciplines can be used to carry out college physical education in multimedia network classrooms, so that the integration between courses can be truly realized, with the aim that all courses can share the excellent results brought by the development of modern education technology. More educators understand, accept, and participate in the practice of college physical education based on multimedia network classrooms and better serve the education of college physical education. The construction of the college physical education evaluation system should be combined with the characteristics of the 5G multimedia network era. The evaluation process includes data collection, data analysis, result output, and result feedback. Each link is an indispensable part of the college physical education evaluation process. Based on the relevant knowledge of the 5G multimedia network, the evaluation indicators determined in this study can basically reflect the various elements of the physical education process in colleges and universities. The distribution of index weight coefficients is more scientific and reasonable. Compared with the current system, the college physical education evaluation system constructed by exploration has a certain degree of objectivity and scientificity. Therefore, it is feasible to apply the 5G multimedia network to the evaluation of college physical education.

Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.

Video hyperlinking represents a classical example of multimodal problems. Common approaches to such problems are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially deep autoencoders, have proven promising both for crossmodal translation and for early fusion via multimodal embedding. A particular architecture, bidirectional symmetrical deep neural networks, have been proven to yield improved multimodal embeddings over classical autoencoders, while also being able to perform crossmodal translation. In this work, we focus firstly at evaluating good single-modal continuous representations both for textual and for visual information. Word2Vec and paragraph vectors are evaluated for representing collections of words, such as parts of automatic transcripts and multiple visual concepts, while different deep convolutional neural networks are evaluated for directly embedding visual information, avoiding the creation of visual concepts. Secondly, we evaluate methods for multimodal fusion and crossmodal translation, with different single-modal pairs, in the task of video hyperlinking. Bidirectional (symmetrical) deep neural networks were shown to successfully tackle downsides of multimodal autoencoders and yield a superior multimodal representation. In this work, we extensively tests them in different settings, with different single-modal representations, within the context of video-hyperlinking. Our novel bidirectional symmetrical deep neural networks are compared to classical autoencoders and are shown to yield significantly improved multimodal embeddings that significantly (alpha=0.0001) outperform multimodal embeddings obtained by deep autoencoders with an absolute improvement in precision at 10 of 14.1% when embedding visual concepts and automatic transcripts and an absolute improvement of 4.3% when embedding automatic transcripts with features obtained with very deep convolutional neural networks, yielding 80% of precision at 10.

Toxicity is paramount for comprehending compound properties, particularly in the early stages of drug design. Due to the diversity and complexity of toxic effects, it became a challenge to compute compound toxicity tasks. To address this issue, we propose a multimodal representation learning model, termed multimodal graph isomorphism network (MMGIN), to address this challenge for compound toxicity multitask learning. Based on fingerprints and molecular graphs of compounds, our MMGIN model incorporates a multimodal representation learning model to acquire a comprehensive compound representation. This model adopts a two-channel structure to independently learn fingerprint representation and molecular graph representation. Subsequently, two feedforward neural networks utilize the learned multimodal compound representation to perform multitask learning, encompassing compound toxicity classification and multiple compound category classification simultaneously. To test the effectiveness of our model, we constructed a novel data set, termed the compound toxicity multitask learning (CTMTL) data set, derived from the TOXRIC data set. We compare our MMGIN model with other representative machine learning and deep learning models on the CTMTL and Tox21 data sets. The experimental results demonstrate significant advancements achieved by our MMGIN model. Furthermore, the ablation study underscores the effectiveness of the introduced fingerprints, molecular graphs, the multimodal representation learning model, and the multitask learning model, showcasing the model's superior predictive capability and robustness.

With the rapid development of social media and the increasing scale of social media data, the rumor detection on social media platforms has become vitally crucial. The key challenges for rumor detection on social media platforms are how to identify rumors deeply entangled with the specific content and how to detect rumors for the emerging social media events without labeled data. Unfortunately, most of the existing approaches can hardly handle these challenges since they tend to learn event-specific features and cannot transfer the learned features to newly emerged events. To tackle the above challenges, we propose a novel Multimodal Disentangled Domain Adaption (MDDA) method which can derive event-invariant features and thus benefit the detection of rumors on emerging social media events. The model consists of two components: the multimodal disentangled representation learning and the unsupervised domain adaptation. The multimodal disentangled representation learning is responsible for disentangling the multimedia posts into the content features and the rumor style features, and removing the content-specific features from post representation. The unsupervised domain adaptation aims to filter out the event-specific features and keep shared rumor style features among events. Based on the final event-invariant rumor style features, we train a robust social media rumor detector that can transfer knowledge from source events to the target events, which can perform well on the newly emerged events. Extensive experiments on two Twitter benchmark datasets demonstrate that our rumor detection model outperforms state-of-the-art methods.

The launching of low-cost depth sensors opens up new potentials for RGB-D-based human action recognition. However, most of current RGB-D-based methods simply fuse multimodal features in a holistic manner and ignore the latent connections among different modalities. In this paper, we propose a multimodal correlative representation learning (MCRL) model for human action recognition from RGB-D videos. Specifically, we propose a spatio-temporal pyramid Fourier HOG feature (STPF-HOG) to capture local dynamic patterns around each human joint, which integrates both spatial arrangement and temporal structures. The proposed MCRL model utilizes multimodal data (skeleton, depth, and RGB), and learns shared structures among different modalities. The original low-level features are effectively compressed and projected to a latent subspace. Then, the discriminative shared features are learned by a supervised fashion. We formulate both subspace learning and shared features mining in a modified multi-task learning framework and solve the formulation using an iterative optimization algorithm. To perform computationally efficient action recognition, a robust collaborative representation classifier is presented by introducing a weight regularization matrix. Experimental results on three action datasets demonstrate that the proposed method leads to a more favorable performance compared with the state-of-the-art methods.

Deep contrastive representation learning for multi-modal clustering

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

International audience

The purpose of this study is to propose an emotion recognition algorithm for piano music based on multimodal learning. Multi-modal learning is a learning method that combines various types of data. By combining different modal information, learning tasks can be completed more comprehensively and accurately. In the field of music emotion recognition, multimodal learning can extract the emotional features of music from different angles, and improve the recognition accuracy and robustness. In this study, the multi-modal learning method is mainly used, and the representative features are extracted by combining the audio signal, score information and the performance of the piano music, and the machine learning algorithm is used to classify the emotions. The audio signal is preprocessed, including noise reduction, cutting and other operations, and then the features of the audio signal are obtained by feature extraction methods, such as short-time Fourier transform (STFT) and formant analysis (RPA). Digitally convert the score, extract the structure, melody, harmony and other characteristics of the music, and obtain the emotional information of the music by analyzing the relationship between symbols and notes in the score. By analyzing the performance skills, strength, speed and other parameters of the player, the characteristics related to the emotional expression of the player are extracted. Machine learning algorithm is used to train and classify the fusion features, and the emotion recognition of piano music is realized. In this study, the emotion recognition algorithm of piano music based on multimodal learning can mine the emotion information of music at different levels, improve the accuracy and robustness of emotion recognition, and provide useful reference for the field of music emotion calculation.

In the rapidly advancing field of multi-modal machine learning (MMML), the convergence of multiple data modalities has the potential to reshape various applications. This paper presents a comprehensive overview of the current state, advancements, and challenges of MMML within the sphere of engineering design. The review begins with a deep dive into five fundamental concepts of MMML: multi-modal information representation, fusion, alignment, translation, and co-learning. Following this, we explore the cutting-edge applications of MMML, placing a particular emphasis on tasks pertinent to engineering design, such as cross-modal synthesis, multi-modal prediction, and cross-modal information retrieval. Through this comprehensive overview, we highlight the inherent challenges in adopting MMML in engineering design, and proffer potential directions for future research. To spur on the continued evolution of MMML in engineering design, we advocate for concentrated efforts to construct extensive multi-modal design datasets, develop effective data-driven MMML techniques tailored to design applications, and enhance the scalability and interpretability of MMML models. MMML models, as the next generation of intelligent design tools, hold a promising future to impact how products are designed.

Multimodal emotion recognition based on facial videos aims to extract features from different modalities to identify human emotions. The previous work focus on designing various fusion schemes to combine heterogeneous modal data. However, most studies have overlooked the role of different modalities in emotion recognition and have not fully utilized the intrinsic connections between modalities. Furthermore, the multimodal data from facial videos also contain various distractions bad for emotion analysis. How to reduce the impact of distractions and enable a model to mine effective information for emotion recognition from different modalities is still a challenge problem. To address above issue, we propose a SVD-guided multimodal feature fusion method based on facial video for emotion recognition, which uses a hierarchical fusion mechanism and adopts different loss strategies at each level to learn multimodal feature representation. Specifically, we fuse the facial expression and rPPG signal (or Point-of-Gaze) by using the weak supervision strategy and contrastive learning. Subsequently, the fused feature of facial expression and rPPG signal and the fused feature of facial expression and Point-of-Gaze are combined together to construct the unified multimodal feature matrix. Based on this, Singular Value Decomposition (SVD) is used to refine the redundancy information caused by the multimodal fusion and guide the neural network to learn discriminative emotion feature. At the same time, a consistent loss is developed to enhance the multimodal representation. Experiments on three public datasets show that the proposed method achieves better results over the compared methods.