Vision-language constraint graph representation learning for unsupervised vehicle re-identification

In this paper, we propose a cross-media regularization framework to enhance image understanding which can benefit image retrieval, classification and so on. The goal of cross-media regularization is to find regularization projections by exploiting the correlations between visual features and textual features. Thus, the original noisy distribution of visual features can be refined by leveraging the discriminative distribution of the corresponding textual features. Within the proposed cross-media regularization framework, a mid-level representation is built by jointly projecting both visual and textual features into a shared feature subspace, which leads to transferring of the discriminative semantic characteristic embedded in the textual modality into the corresponding visual modality. Meanwhile, the discriminative characteristic of textual features can also be boosted simultaneously. The experimental results demonstrate that the proposed mid-level space learning process can remarkably improve the search quality and outperform the existing semantic regularization methods.

A feature-based restoration dynamic interaction network for multimodal sentiment analysis

The associations between different modalities of Web images could be very useful for Web image retrieval. In this paper, we investigate the multi-modal associations between two basic modalities of web images, i.e. keyword and visual feature clusters, by data mining technique. The association rule crosses two modalities, in which the antecedent is a single keyword and the consequent is several visual feature clusters. A customized mining process is provided to mine such special multi-modal association rules. The multi-modal association rules are obtained offline based on the existing inverted file and utilized online to automatically integrate the keyword and visual features for web image retrieval. The experiments are carried out in a prototype system for Web image retrieval, and the results show the effectiveness of the mined multi-modal association rules.

Cross-modality integration framework with prediction, perception and discrimination for video anomaly detection

The aim of this paper is to categorize movies into genres using the previews. Our study attempts to combine audio, visual and text features to classify a collection of movie previews into action, biography, comedy, and horror. For each of the collected previews, the audio and visual features are extracted and the text features are drawn from social tags via social websites. The probabilistic latent semantic analysis (PLSA) is used to incorporate the features from these three different aspects of information. The standard PLSA processes one type of information only. Therefore double-model and triple-model PLSAs are extended in order to combine two or three different types of information. We compare these various variants of PLSA approaches with unimodal PLSAs, which use either audio, visual or text features only. The experimental results show not only that one of the triple-model PLSAs achieves the highest accuracy, but also that social tags (text features) play an important role for classifying movies genres.

Fake news that combines text and images has a better story-telling ability than text-only fake news, making it more deceptive and easier to spread maliciously. Therefore, multi-modal fake news detection has become a new hot topic. There are two main challenges in this task. First, the traditional pre-training model BERT has more parameters and a relatively slow training speed, which limits the level of extracted text features. Second, in the multi-modal form, the fusion process is only a simple splicing of visual and textual features of news, and the obtained multi-modal features are insufficient to express the complementarity between multi-modal data and may have redundant information, potentially leading to biased detection results.In order to solve the above issues, we propose ALB-MCF, using ALBERT and Multi-modal Circulant Fusion(MCF) for fake news detection. ALB-MCF consists of four main modules: a multi-modal feature extractor, a multi-modal feature fusion, a fake news detection and a domain classifier. Specifically, the multi-modal feature extractor innovatively uses a pre-trained ALBERT model to extract text features and a pre-trained VGG19 model to extract visual features. Then, the text features and the visual features are fused into a multi-modal feature representation by MCF, which improves the fusion capability while avoiding an increase in parameters and computational cost. Finally, the multi-modal features are fed into the detector to detect fake news. The role of the domain classifier is mainly to remove event-specific features while retaining shared features between events, thus providing effective detection of emerging events. We have conducted extensive experiments on two real-world datasets. The results demonstrated that our model can handle multi-modal data more effectively, thus improving the accuracy of fake news detection.

Affective analysis of images in social networks has drawn much attention, and the texts surrounding images are proven to provide valuable semantic meanings about image content, which can hardly be represented by low-level visual features. In this paper, we propose a novel approach for visual affective classification (VAC) task. This approach combines visual representations along with novel text features through a fusion scheme based on Dempster-Shafer (D-S) Evidence Theory. Specifically, we not only investigate different types of visual features and fusion methods for VAC, but also propose textual features to effectively capture emotional semantics from the short text associated to images based on word similarity. Experiments are conducted on three public available databases: the International Affective Picture System (IAPS), the Artistic Photos and the MirFlickr Affect set. The results demonstrate that the proposed approach combining visual and textual features provides promising results for VAC task.


 
 
 The perceived quality of a document is affected by various factors, including grammat- icality, readability, stylistics, and expertise depth, making the task of document quality assessment a complex one. In this paper, we explore this task in the context of assessing the quality of Wikipedia articles and academic papers. Observing that the visual rendering of a document can capture implicit quality indicators that are not present in the document text — such as images, font choices, and visual layout — we propose a joint model that combines the text content with a visual rendering of the document for document qual- ity assessment. Our joint model achieves state-of-the-art results over five datasets in two domains (Wikipedia and academic papers), which demonstrates the complementarity of textual and visual features, and the general applicability of our model. To examine what kinds of features our model has learned, we further train our model in a multi-task learning setting, where document quality assessment is the primary task and feature learning is an auxiliary task. Experimental results show that visual embeddings are better at learning structural features while textual embeddings are better at learning readability scores, which further verifies the complementarity of visual and textual features.
 
 

Chapter 21 - Optical flow-based representation for video action detection

As one of the most popular social media platforms, microblogs are ideal places for news propagation. In microblogs, tweets with both text and images are more likely to attract attention than text-only tweets. This advantage is exploited by fake news producers to publish fake news, which has a devasting impact on individuals and society. Thus, multimodal fake news detection has attracted the attention of many researchers. For news with text and image, multimodal fake news detection utilizes both text and image information to determine the authenticity of news. Most of the existing methods for multimodal fake news detection obtain a joint representation by simply concatenating a vector representation of the text and a visual representation of the image, which ignores the dependencies between them. Although there are a small number of approaches that use the attention mechanism to fuse them, they are not fine-grained enough in feature fusion. The reason is that, for a given image, there are multiple visual features and certain correlations between these features. They do not use multiple feature vectors representing different visual features to fuse with textual features, and ignore the correlations, resulting in inadequate fusion of textual features and visual features. In this paper, we propose a novel fine-grained multimodal fusion network (FMFN) to fully fuse textual features and visual features for fake news detection. Scaled dot-product attention is utilized to fuse word embeddings of words in the text and multiple feature vectors representing different features of the image, which not only considers the correlations between different visual features but also better captures the dependencies between textual features and visual features. We conduct extensive experiments on a public Weibo dataset. Our approach achieves competitive results compared with other methods for fusing visual representation and text representation, which demonstrates that the joint representation learned by the FMFN (which fuses multiple visual features and multiple textual features) is better than the joint representation obtained by fusing a visual representation and a text representation in determining fake news.

This paper presents a correlation-aware attention branch network (CorABN) using multi-modal data for deterioration level estimation of infrastructures. CorABN can collaboratively use visual features from distress images and text features from text data recorded at the inspection to improve the estimation performance of deterioration levels. Specifically, by maximizing correlation between the visual and text features that provide useful information for the deterioration level estimation, a correlation-aware attention map can be generated. Besides, the text features are also utilized in the final estimation along with visual features improved by the attention mechanism to achieve higher estimation performance. With the losses based on both the estimation performance and the correlation, CorABN can train the entire model in an end-to-end manner. Experiments using distress images and their corresponding text data show the effectiveness of the proposed method.

Video emotion recognition aims to infer human emotional states from the audio, visual, and text modalities. Previous approaches are centered around designing sophisticated fusion mechanisms, but usually ignore the fact that text contains global semantic information, while speech and face video show more fine-grained temporal dynamics of emotion. From the perspective of cognitive sciences, the process of emotion expression, either through facial expression or speech, is implicitly regulated by high-level semantics. Inspired by this fact, we propose a multimodal interaction enhanced representation learning framework for emotion recognition from face video, where a semantic enhancement module is first designed to guide the audio/visual encoder using the semantic information from text, then the multimodal bottleneck Transformer is adopted to further reinforce the audio and visual representations by modeling the cross-modal dynamic interactions between the two feature sequences. Experimental results on two benchmark emotion databases indicate the superiority of our proposed method. With the semantic enhanced audio and visual features, it outperforms the state-of-the-art models which fuse the features or decisions from the audio, visual and text modalities.

Advances in healthcare social media and information about the doctor-patient (D–P) communication regarding the prior patients’ treatment experience, can positively influence the D–P relationship. In pace with prior patients’ photo-sharing on healthcare social media websites from personal computers and smartphones regarding their treatment experience, the amount of multi-modal content has been growing exponentially. Therefore, there is an increasing need for coping with such information to mine useful knowledge about the D–P communication. Scraping 68,610 reviews, including 4618 photos from a popular physician-rating site, Yelp.com , this study proposes a novel, real-time, multi-modal classification framework, which uses textual and visual modalities as a source of information. Furthermore, this work suggests a social media image filtering mechanism that filters duplicate and irrelevant information from the data. Results show that the data filtering enhances the information reliability, whereas the addition of novel text and visual feature sets improves the classification accuracy up to 16.94%. In addition, fusing textual and visual features enhance the performance of the classifier by 18.24%, which produces better results than considering them separately. The findings also revealed that deep learning algorithms outperformed the classical machine learning algorithms across the entire novel features model, indicating the usefulness and suitability of the proposed methodology. Lastly, the findings from extensive experiments on the physicians’ reviews dataset will guide the doctors to demonstrate the implication of the proposed system for improving the D–P relationship.

Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this article, we deal with the fast video temporal grounding (FVTG) task, aiming at localizing the target segment with high speed and favorable accuracy. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance, which suffer from the test-time bottleneck. Although several common space-based methods enjoy the high-speed merit during inference, they can hardly capture the comprehensive and explicit relations between visual and textual modalities. In this article, to tackle the dilemma of the speed–accuracy tradeoff, we propose a commonsense-aware cross-modal alignment network (C 2 AN) that incorporates commonsense-guided visual and text representations into a complementary common space for fast video temporal grounding. Specifically, the commonsense concepts are explored and exploited by extracting the structural semantic information from a language corpus. Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts. Finally, to maintain the original semantic information of textual queries, a cross-modal complementary common space is optimized to obtain matching scores for performing FVTG. Extensive results on two challenging benchmarks show that our C 2 AN method performs favorably against states of the art while running at high speed. Our code is available at https://github.com/ZiyueWu59/CCA

The Dense Audio-Visual Event Localization (DAVEL) task aims to temporally localize events in untrimmed videos that occur simultaneously in both the audio and visual modalities. This paper explores DAVEL under a new and more challenging weakly-supervised setting (W-DAVEL task), where only video-level event labels are provided and the temporal boundaries of each event are unknown. We address W-DAVEL by exploiting cross-modal salient anchors, which are defined as reliable timestamps that are well predicted under weak supervision and exhibit highly consistent event semantics across audio and visual modalities. Specifically, we propose a Mutual Event Agreement Evaluation module, which generates an agreement score by measuring the discrepancy between the predicted audio and visual event classes. Then, the agreement score is utilized in a Cross-modal Salient Anchor Identification module, which identifies the audio and visual anchor features through global-video and local temporal window identification mechanisms. The anchor features after multimodal integration are fed into an Anchor-based Temporal Propagation module to enhance event semantic encoding in the original temporal audio and visual features, facilitating better temporal localization under weak supervision. We establish benchmarks for W-DAVEL on both the UnAV-100 and ActivityNet1.3 datasets. Extensive experiments demonstrate that our method achieves state-of-the-art performance.