Intentonomy: a Dataset and Study towards Human Intent Understanding

CE-DCVSI: Multimodal relational extraction based on collaborative enhancement of dual-channel visual semantic information

This paper presents a multiple feature fusion method using topic model for social image visualization. Images in social media are represented from several aspects such as their visual information and tags. The proposed method extracts low-level features from social images and their tags and calculates their integrated high-level features. Specifically, the proposed method applies multilayer multimodal probabilistic Latent Semantic Analysis (mm-pLSA) to the low-level visual and tag features to obtain the high-level features. Then, by applying dimensionality reduction techniques to the obtained features, successful visualization becomes feasible.

Recent years have witnessed a great success of social media websites. Tag-based image search is an important approach to access the image content of interest on these websites. However, the existing ranking methods for tag-based image search frequently return results that are irrelevant or lacking in diversity. Most of the existing methods estimate the relevance of images by using tags and visual characteristics either separately or sequentially. The proposed system uses an approach that utilize simultaneously both visual information and textual information in real time to estimate the relevance of user tagged image. The method used to determine the relevance estimation is the hypergraph learning approach. The hypergraph is a generalization of a graph in which an edge in the hypergraph can be connected to any number of vertices. In the proposed method each social image can be represented by the bag-of-visual words and bag-of-textual words, which can be obtained from the textual content and visual content of the particular image. A hypergraph can be constructed in which the vertices represent the social images for ranking and the each hyperedge represents the visual words or tags that are obtained from the image. In the hypergraph learning scheme, both the visual content and tag information are taken into consideration at same time. Different from the method used by the traditional hypergraph, in the proposed system a social image hypergraph is constructed where vertices represent the images and hyperedges represent the visual or textual terms. The set of pseudo-positive images are used to achieve the learning, where the weight of hyperedges are updated throughout the learning process. Thus only the most relevant images are given to the user.

Research in the Vision and Language area encompasses challenging topics that seek to connect visual and textual information. When the visual information is related to videos, this takes us into Video-Text Research, which includes several challenging tasks such as video question answering, video summarization with natural language, and video-to-text and text-to-video conversion. This paper reviews the video-to-text problem, in which the goal is to associate an input video with its textual description. This association can be mainly made by retrieving the most relevant descriptions from a corpus or generating a new one given a context video. These two ways represent essential tasks for Computer Vision and Natural Language Processing communities, called text retrieval from video task and video captioning/description task. These two tasks are substantially more complex than predicting or retrieving a single sentence from an image. The spatiotemporal information present in videos introduces diversity and complexity regarding the visual content and the structure of associated language descriptions. This review categorizes and describes the state-of-the-art techniques for the video-to-text problem. It covers the main video-to-text methods and the ways to evaluate their performance. We analyze twenty-six benchmark datasets, showing their drawbacks and strengths for the problem requirements. We also show the progress that researchers have made on each dataset, we cover the challenges in the field, and we discuss future research directions.

As mobile and chatbot-based shopping platforms continue to reshape online retail experiences, consumers are increasingly exposed to product information in sequential rather than simultaneous formats. Drawing on information foraging theory, we examine the marginal value consumers derive when textual and visual product information are presented sequentially. We conducted an online experiment (n = 86) with student sample to test whether different information presentation sequence (text first vs. image first) would lead to different perceived diagnosticity, product attitude, and purchase intention. We also examined whether this sequence effect differed between search (notebook and hanger) and experience (sofa and portable closet) products. Findings from mixed-effects regressions show that presenting textual information before visual information results in greater marginal gains in all three outcomes, particularly for experience products. These findings offer new insights into the cognitive dynamics of sequential information processing and practical implications for optimizing product presentation in online retail design.

Recent years have seen a rapid growth in the number of fake news that are posted online. Fake news detection is very challenging since they are usually created to contain a mixture of false and real information and images that have been manipulated that confuses the readers. In this paper, we propose a multimodal system with the aim to differentiate between fake and real posts. Our system is based on a neural network and combines textual, visual and semantic information. The textual information is extracted from the content of the post, the visual one from the image that is associated with the post and the semantic refers to the similarity between the image and the text of the post. We conduct our experiments on three standard real world collections and we show the importance of those features on detecting fake news.

Medical image retrieval can assist physicians in finding information supporting their diagnosis and fulfilling information needs. Systems that allow searching for medical images need to provide tools for quick and easy navigation and query refinement as the time available for information search is often short. Relevance feedback is a powerful tool in information retrieval. This study evaluates relevance feedback techniques with regard to the content they use. A novel relevance feedback technique that uses both text and visual information of the results is proposed. The two information modalities from the image examples are fused either at the feature level using the Rocchio algorithm or at the query list fusion step using a common late fusion rule. Results using the ImageCLEF 2012 benchmark database for medical image retrieval show the potential of relevance feedback techniques in medical image retrieval. The mean average precision (mAP) is used as the evaluation metric and the proposed method outperforms commonly-used methods. The baseline without feedback reached 16 % whereas the relevance feedback with 20 images reached up to 26.35 % with three steps and when using 100 images up to 34.87 % in four steps. Most improvements occur in the first two steps of relevance feedback and then results start to become relatively flat. This might also be due to only using positive feedback as negative feeback often also improves results after more steps. The effect of relevance feedback in automatically spelling corrected and translated queries is investigated as well. Results without mistakes were better than spell-corrected results but the spelling correction more than double results over non-corrected retrieval. Multimodal relevance feedback has shown to be able to help visual medical information retrieval. Next steps include integrating semantics into relevance feedback techniques to benefit from the structured knowledge of ontologies and experimenting on the fusion of text and visual information.

Visual information had been found to be superior to textual information in enhancing processing ease, preference, attitude, and purchase intention. However, these visual format’s superior effects were mainly established for hedonic products (e.g. jackets, crackers) without a clear measure of objective decision quality. In complex decision environments (e.g. healthcare and credit card), it is unclear whether the visual advantage persists – especially when objective decision quality is a more important outcome than attitude or behavioral intentions. This research addresses this gap and examines the effect of visual vs. textual information on measures of objective decision quality in three different decision tasks (i.e. health insurance, credit card, jacket). The empirical findings suggest that the visual advantage does not extend to objective decision quality as measured by (i) the proportion to choose a dominant alternative within a choice task, and (ii) the proportion to remain with the chosen alternative when a change is allowed. Additionally, the findings show that an individual’s confidence in his/her chosen alternative was higher in the visual than textual condition for the jacket but not for health insurance and credit card decision tasks. The findings also provide evidence supporting the visual advantage effects on subjective measures of perceived time spent on task, attitude, and purchase intention.

Sentiment-aware multimodal pre-training for multimodal sentiment analysis

The rapid growth of edge data generated by mobile devices and applications deployed at the edge of the network has exacerbated the problem of information overload. As an effective way to alleviate information overload, recommender system can improve the quality of various services by adding application data generated by users on edge devices, such as visual and textual information, on the basis of sparse rating data. The visual information in the movie trailer is a significant part of the movie recommender system. However, due to the complexity of visual information extraction, data sparsity cannot be remarkably alleviated by merely using the rough visual features to improve the rating prediction accuracy. Fortunately, the convolutional neural network can be used to extract the visual features precisely. Therefore, the end-to-end neural image caption (NIC) model can be utilized to obtain the textual information describing the visual features of movie trailers. This paper proposes a trailer inception probabilistic matrix factorization model called Ti-PMF, which combines NIC, recurrent convolutional neural network, and probabilistic matrix factorization models as the rating prediction model. We implement the proposed Ti-PMF model with extensive experiments on three real-world datasets to validate its effectiveness. The experimental results illustrate that the proposed Ti-PMF outperforms the existing ones.

In recent years, large amounts of information have been transmitted through SNSs. Mainstream SNSs focus mainly on images. Given that many individuals use SNSs to express their sentiments, understanding the individual feelings of such users can be useful for helping identify potential demand in the society. Sentiment analysis of images in SNSs has thus attracted research attention. However, many existing studies have utilized a supervised learning approach with a high cost for labeling. By contrast, Wang et al., who used unsupervised learning, have acquired the sentiment of images through matrix decomposition from text information and visual information. However, text and visual information are not sufficient to acquire the desired result. In addition, sentiment polarity of the image is greatly influenced by text information. Therefore, this work will conduct a sentiment analysis of images by using tensor decomposition with an unsupervised learning approach. By considering the sentiment polarity of both text and visual features, this work aims to avoid being influenced only by text information.

A new medical image retrieval approach based on low-level features and high-level semantic features is proposed. The image is segmented into several sub-images by fuzzy C-mean clustering algorithm. After extracting the three gray features, in order to emphasize shape and texture features, the sub-image is changed to binary image, and seven shape features and four texture features are extracted. To decrease feature vector dimension, genetic algorithm is used. The optimizing features constitute the visual contents. Only using these low-level features can't get satisfying results, because these features can't entirely describe human recognition to images, the semantic features are needed, but it is difficult to extract the image semantic contents directly at the present time, so the text information in the image report by radiologists is chosen for semantic content. Experimental results show that the retrieval integrating low-level visual features and text information is better than that only by visual contents.

Intent recognition, as a core technology of human-computer interaction systems, directly affects the quality of intelligent services. Traditional single-modal intent recognition is limited by the constraints of a single information source, resulting in low accuracy in complex scenarios. This paper proposes a Cross-Modal Attention Fusion (CMAF) model for multimodal intent recognition, which achieves accurate intent recognition through deep fusion of text, audio, and visual multimodal information. The model adopts a hierarchical feature extraction architecture and combines intra-modal attention with cross-modal attention mechanisms to effectively capture semantic correlations and complementary information between modalities. Experimental results on three public datasets, namely MIntRec, SIMMC, and the newly added medical consultation dataset (MedDialog), show that the CMAF model improves the accuracy by 4.2%, 3.8%, and 5.1% respectively, and the macro-F1 score by 3.9%, 3.5%, and 4.7% respectively compared with baseline models. Particularly, the performance degradation of CMAF is only 5.2% under 0dB noise, verifying the significant advantages of multimodal fusion in intent recognition tasks.

Social interactions rely on the interpretation of semantic and emotional information, often from multiple sensory modalities. Nonhuman primates send and receive auditory and visual communicative signals. However, the neural mechanisms underlying the association of visual and auditory information based on their common social meaning are unknown. Using heart rate estimates and functional neuroimaging, we show that in the lateral and superior temporal sulcus of the macaque monkey, neural responses are enhanced in response to species-specific vocalisations paired with a matching visual context, or when vocalisations follow, in time, visual information, but inhibited when vocalisation are incongruent with the visual context. For example, responses to affiliative vocalisations are enhanced when paired with affiliative contexts but inhibited when paired with aggressive or escape contexts. Overall, we propose that the identified neural network represents social meaning irrespective of sensory modality.

The principal concern of this chapter is to provide those in the visual information retrieval community with a methodology which allows them to integrate the results of content analysis of visual information, i.e., the content descriptors, and their text-based representation to attain the semantically precise results of keyword-based image retrieval operations. The main visual objects of our discussion are images which do not have any semantic representations therein. Those images demand textual annotation of precise semantics which is to be based on the results of automatic content analysis but not on the results of time-consuming manual annotation processes. We first outline the technical background and literature review on a variety of annotation techniques for visual information retrieval. We then describe our proposed method and its implemented system for generating metadata or textual indexes to visual objects by using content analysis technique by bridging the gaps between content descriptors and textual information.