Joint Image-Text Representation by Gaussian Visual-Semantic Embedding

Hyperspectral images (HSIs) provide invaluable information in both spectral and spatial domains for image classification tasks. In this paper, we use semantic representation as a middle-level feature to describe image pixels’ characteristics. Deriving effective semantic representation is critical for achieving good classification performance. Since different image descriptors depict characteristics from different perspectives, combining multiple features in the same semantic space makes semantic representation more meaningful. First, a probabilistic support vector machine is used to generate semantic representation-based multifeatures. In order to derive better semantic representation, we introduce a new adaptive spatial regularizer that well exploits the local spatial information, while a nonlocal regularizer is also used to search for global patch-pair similarities in the whole image. We combine multiple features with local and nonlocal spatial constraints using an extended Markov random field model in the semantic space. Experimental results on three hyperspectral data sets show that the proposed method provides better performance than several state-of-the-art techniques in terms of region uniformity, overall accuracy, average accuracy, and Kappa statistics.

In image-text cross-modal retrieval, most existing methods map the samples to point representations that relate the given sample to the specific single point and do not reflect the correlations between the given sample and all the points in semantic space, so that the semantic complexities of the samples and the local similarities between the samples are not fully exploited. To address the above issues, a probability distribution representation learning approach for image-text cross-modal retrieval is presented. Specifically, the method first incorporates lots of label information to capture the prominent features of the samples, which guides the model to construct the semantic spaces for different modalities and learn the distributions of the samples, based on variational information bottleneck. The probability densities of the learned distribution naturally reflect the correlations between the corresponding sample and all the points in semantic space. Moreover, the hinge triplet loss is introduced to align the distributions of the samples from different modalities at the semantic level, which makes the similar image-text pair have similar distributions. Finally, the learned distributions are used to represent the corresponding samples and the Bhattacharyya distance of the learned distributions to measure the similarities between the samples is leveraged, which is able to model the semantic complexities and explore the local similarities. The results of the experiments on Wikipedia and Pascal Sentence demonstrate that proposed method outperforms all of the 9 compared methods. The average mAP for the retrieval tasks of image to text and text to image evenly improves 15.0 percent points on Wikipedia and 13.6 percent points on Pascal Sentence.

Nowadays online image search become more essential. In this paper, we have extended existing system for image re-ranking is explained. The existing system is divided into offline and online parts. In offline part various semantic spaces are automatically learns for different query keywords. Image Semantic content as signatures are generated by mapping the image features i.e. visual features into its semantic spaces related to image context. In online stage, semantic signatures computed from the different semantic space mentioned by the query keyword are equated with semantic signatures of query image for image re-ranking. We are extended the current frame work by adding new technique of hashing. Semantic signatures are small in dimensions, it is possible to make it more compressed and with use of hashing technologies it further enhance their matching efficiency. In this we use locality sensitive hashing concept based on nearest neighbor algorithms. To find more similar item in d-dimensional space, these algorithms are already been applied in different practical scenarios. In this, we implemented a recently discovered hashing-based algorithm to improve the online matching effectiveness of image re-ranking system, for the case the images are represented as objects as points in the rf-dimensional Euclidean space. The locality sensitive hashing algorithm produces the output which is optimal near in the class of nearest neighbor algorithms. The online matching efficiency is improved by using the hashing technique as compare to existing search methods. With the use of hashing technique the system performance is improved by 38%.

In order to improve the robustness of the similarity metric method of image classification, and reduce the complexity of the measure function, the Pearson correlation coefficient is introduced to improve the zero-shot image classification. Firstly, the mapping matrix from visual space to semantic space is learned by the training dataset, and the visual feature is aligned by it. Then in the semantic space, the similarity between features is calculated by the metric function, predicting the label of unseen class by the nearest neighbours. Experiments show that zero-shot image classification based on Pearson correlation coefficient is better than Euclidean distance and cosine similarity.

Zero-shot learning (ZSL) aims to classify unseen samples based on the relationship between the learned visual features and semantic features. Traditional ZSL methods typically capture the underlying multimodal data structures by learning an embedding function between the visual space and the semantic space with the Euclidean metric. However, these models suffer from the hubness problem and domain bias problem, which leads to unsatisfactory performance, especially in the generalized ZSL (GZSL) task. To tackle such a problem, we formulate a discriminative cross-aligned variational autoencoder (DCA-VAE) for ZSL. The proposed model effectively utilizes a modified cross-modal-alignment variational autoencoder (VAE) to transform both visual features and semantic features obtained by the discriminative cosine metric into latent features. The key to our method is that we collect principal discriminative information from visual and semantic features to construct latent features which contain the discriminative multimodal information associated with unseen samples. Finally, the proposed model DCA-VAE is validated on six benchmarks including the large dataset ImageNet, and several experimental results demonstrate the superiority of DCA-VAE over most existing embedding or generative ZSL models on the standard ZSL and the more realistic GZSL tasks.

Graph convolutional networks (GCNs) have achieved great success in many applications and have caught significant attention in both academic and industrial domains. However, repeatedly employing graph convolutional layers would render the node embeddings indistinguishable. For the sake of avoiding oversmoothing, most GCN-based models are restricted in a shallow architecture. Therefore, the expressive power of these models is insufficient since they ignore information beyond local neighborhoods. Furthermore, existing methods either do not consider the semantics from high-order local structures or neglect the node homophily (i.e., node similarity), which severely limits the performance of the model. In this article, we take above problems into consideration and propose a novel Semantics and Homophily preserving Network Embedding (SHNE) model. In particular, SHNE leverages higher order connectivity patterns to capture structural semantics. To exploit node homophily, SHNE utilizes both structural and feature similarity to discover potential correlated neighbors for each node from the whole graph; thus, distant but informative nodes can also contribute to the model. Moreover, with the proposed dual-attention mechanisms, SHNE learns comprehensive embeddings with additional information from various semantic spaces. Furthermore, we also design a semantic regularizer to improve the quality of the combined representation. Extensive experiments demonstrate that SHNE outperforms state-of-the-art methods on benchmark datasets.

Researchers explored the cognitive process of students by analyzing their eye-tracking data in some specified computer programming tasks. Although there are some quantified analysis and data visualization methods proposed in this area, none of them is designed to visualize the attention migration in debugging task. In this paper, we designed a novel kind of chart, i.e. Attention Cloud Map (ACM), for this purpose. We model the semantic code space into a polar coordinate system, and map the code lines and functions of the test program are mapped into points and sectors. We put the bug line in the center point, and mark the relevant code areas in the code space. By plotting the heat map of the amount of eye fixation attention in given time slices, we can obtain a serial of attention distributions in the code space. Experiment results show that, such attention cloud map can enable us to observe the debugging strategy of individual student, and abstract the average progress of one student group, which can be provide some references for the evaluation of students’ debugging performances.

This study addresses the relation between morphological productivity and semantic transparency. Using distributional semantics, we compare German word formation using particles with derivational word formation. We observed that derivational suffixes, but not particles, tend to make strong independent semantic contributions to their carrier words. In two-dimensional t-SNE maps, complex words show clustering by affix, but not by particle. Furthermore, the semantic vectors of suffixed words are predictable from their base words with higher accuracy than is possible for particle verbs. For particle verbs, but not affixed verbs, semantic similarity within the set of complex words correlated negatively with the number of types. Furthermore, only for particle verbs, a greater number of observed types predicted a reduced probability of observing unseen types. We propose that particle verbs primarily serve the onomasiological function of labeling, resulting in relatively idiosyncratic semantic vectors. By contrast, words sharing derivational affixes form distinct clusters in semantic space while maintaining strong and consistent semantic relations with their base words. This enables these words to serve not only as labels, but also allows them to be used with an anaphoric function in discourse.

One of the main challenges in medical image retrieval is the increasing volume of image data, which render it difficult for domain experts to find relevant information from large data sets. Effective and efficient medical image retrieval systems are required to better manage medical image information. Text‐based image retrieval (TBIR) was very successful in retrieving images with textual descriptions. Several TBIR approaches rely on models based on bag‐of‐words approaches, in which the image retrieval problem turns into one of standard text‐based information retrieval; where the meanings and values of specific medical entities in the text and metadata are ignored in the image representation and retrieval process. However, we believe that TBIR should extract specific medical entities and terms and then exploit these elements to achieve better image retrieval results. Therefore, we propose a novel reranking method based on medical‐image‐dependent features. These features are manually selected by a medical expert from imaging modalities and medical terminology. First, we represent queries and images using only medical‐image‐dependent features such as image modality and image scale. Second, we exploit the defined features in a new reranking method for medical image retrieval. Our motivation is the large influence of image modality in medical image retrieval and its impact on image‐relevance scores. To evaluate our approach, we performed a series of experiments on the medical ImageCLEF data sets from 2009 to 2013. The BM25 model, a language model, and an image‐relevance feedback model are used as baselines to evaluate our approach. The experimental results show that compared to the BM25 model, the proposed model significantly enhances image retrieval performance. We also compared our approach with other state‐of‐the‐art approaches and show that our approach performs comparably to those of the top three runs in the official ImageCLEF competition.

Extracting more information from feature interactions is essential to improve click-through rate (CTR) prediction accuracy. Although deep learning technology can help capture high-order feature interactions, the combination of features lacks interpretability. In this paper, we propose a multi-semantic feature interaction learning network (MeFiNet), which utilizes convolution operations to map feature interactions to multi-semantic spaces to improve their expressive ability and uses an improved Squeeze & Excitation method based on SENet to learn the importance of these interactions in different semantic spaces. The Squeeze operation helps to obtain the global importance distribution of semantic spaces, and the Excitation operation helps to dynamically re-assign the weights of semantic features so that both semantic diversity and feature diversity are considered in the model. The generated multi-semantic feature interactions are concatenated with the original feature embeddings and input into a deep learning network. Experiments on three public datasets demonstrate the effectiveness of the proposed model. Compared with state-of-the-art methods, the model achieves excellent performance (+0.18% in AUC and -0.34% in LogLoss VS DeepFM; +0.19% in AUC and -0.33% in LogLoss VS FiBiNet).

In defense of spatial models of semantic representation

BCLNet: Boundary contrastive learning with gated attention feature fusion and multi-branch spatial-channel reconstruction for land use classification

Analyzing textual semantics in matching user query and service description is critical for Web service discovery. Existing works mostly extract the features of the description and query independently, downgrading them into word-level calculation, which can not jointly extract the accurate semantics. For this issue, this work explores a way to enable the semantic matching for the contents (including words, phrases, or sentences) in query and the sentences in service description, by mapping words and sentences into the same semantic space. Specifically, we propose an unsupervised Bayesian probabilistic model, bi-Directional Hybrid Priors Topic Model (bi-HPTM), to capture the textual semantics of the words and sentences in a probabilistic simplex, which provides a flexible operation to build the semantic links from the queries to service descriptions. Meanwhile, the textual semantics generated by bi-HPTM is highly interpretable that help to understand the user requirements. The proposed model is examined by ProgrammableWeb. Experimental results demonstrate that bi-HPTM outperforms state-of-the-art methods for semantic service discovery on service classification and retrieval. The visualizations of the nearest-neighbored queries and descriptions show the insights of our model on capturing the latent semantics of Web services.

Taking advantage of the large scale corpus on the web to effectively and efficiently mine the topics within texts is an essential problem in the era of big data. We focus on the problem of learning text topic embedding in an unsupervised manner, which enjoys the properties of efficiency and scalability. Text topic embedding represents words and documents in a semantic topic space, in which the words and documents with similar topic will be embedded close to each other. When compared with conventional topic models, which implicitly capture the document-level word co-occurrence patterns, text topic embedding alleviates the data sparsity problem and captures the semantic relevance between different words and documents. To model text topic embedding, we propose a Bidirectional Hierarchical Skip-Gram model (BHSG) based on skip-gram model. BHSG includes two components: semantic generation module to learn semantic relevance between texts and topic enhance module to produce the text topic embedding based on text embedding learned in the former module. We evaluated our method on two kinds of topic-related tasks: text classification and information retrieval. The experimental results on four public datasets and one dataset we provide all demonstrate that our proposed method can achieve a better performance.

We propose a method to study the variation lying between different word embeddings models trained with different parameters. We explore the variation between models trained with only one varying parameter by observing the distributional neighbors variation and show how changing only one parameter can have a massive impact on a given semantic space. We show that the variation is not affecting all words of the semantic space equally. Variation is influenced by parameters such as setting a parameter to its minimum or maximum value but it also depends on the corpus intrinsic features such as the frequency of a word. We identify semantic classes of words remaining stable across the models trained and specific words having high variation.