Scene Graph Knowledge Enhanced Hashing with Contrastive Learning for Image-Text Retrieval

This paper presents a framework for jointly grounding objects that follow certain semantic relationship constraints given in a scene graph. A typical natural scene contains several objects, often exhibiting visual relationships of varied complexities between them. These inter-object relationships provide strong contextual cues towards improving grounding performance compared to a traditional object query-only-based localization task. A scene graph is an efficient and structured way to represent all the objects and their semantic relationships in the image. In an attempt towards bridging these two modalities representing scenes and utilizing contextual information for improving object localization, we rigorously study the problem of grounding scene graphs on natural images. To this end, we propose a novel graph neural network-based approach referred to as Visio-Lingual Message Passing Graph Neural Network (VL-MPAG Net). In VL-MPAG Net, we first construct a directed graph with object proposals as nodes and an edge between a pair of nodes representing a plausible relation between them. Then a three-step inter-graph and intra-graph message passing is performed to learn the context- dependent representation of the proposals and query objects. These object representations are used to score the proposals to generate object localization. The proposed method significantly outperforms the baselines on four public datasets.

TechWordNet: Development of semantic relation for technology information analysis using F-term and natural language processing

Scene graph generation denotes the process of parsing a visual input into a graph representation for downstream reasoning tasks. Most existing research models input scenes as flat scene graphs, capturing solely on horizontal relationships between two objects, while disregarding vertical dependencies. Such flat scene graphs merely describe direct relationships between objects, and cannot represent higher-level semantics. The lack of hierarchical relationships also limits the performance of scene graphs in downstream reasoning tasks. In this work, we propose hierarchical scene graph generation, a new problem task that requires the model to generate scene graphs with clear hierarchy. To achieve this goal, a hierarchical scene graph generation with coarse-to-fine reasoning framework is proposed. It contains three stages: first, summarize the main caption of the scene; second, describe the main visual elements related to the theme; and finally, add secondary elements with background. To facilitate training, a high-quality hierarchical scene graph dataset that comprises 40 K image-scene description pairs is constructed. Utilize rich knowledge and powerful multi-turn conversation capabilities of multi-modal large language models, the proposed framework can not only achieve better semantic understanding and relational modeling, but also seamlessly integrating its relational modeling capabilities to enhance visual-language tasks. Extensive experimental results demonstrate that our method constructs an effective scene representation, outperforming the state-of-the-art by 3.35% in recall while generating significantly fewer average triplets 8.7vs. 9.2. The results on downstream tasks also indicate that hierarchical scene graph significantly contributes to visual-language interactive reasoning.

Learning to transfer focus of graph neural network for scene graph parsing

Cross-modal attention mechanisms have been widely applied to the image-text matching task. They have achieved remarkable improvements thanks to their capability of learning fine-grained relevance across different modalities. However, the cross-modal attention models of existing methods could be sub-optimal and inaccurate because there is no direct supervision provided during the training process. In this work, we propose two novel training strategies, namely Contrastive Content Resourcing (CCR) and Contrastive Content Swapping (CCS) constraints, to address such limitations. These constraints supervise the training of cross-modal attention models in a contrastive learning manner without requiring explicit attention annotations. They are plug-in training strategies and can be generally integrated into existing cross-modal attention models. Additionally, we introduce three metrics, including Attention Precision, Recall, and F1-Score, to quantitatively measure the quality of learned attention models. We evaluate the proposed constraints by incorporating them into four state- of-the-art cross-modal attention-based image-text matching models. Experimental results on both Flickr30k and MS-COCO datasets demonstrate that integrating these constraints generally improves the model performance in terms of both retrieval performance and attention metrics.

To semantically understand remote sensing images, it is not only necessary to detect the objects in them but also to recognize the semantic relationships between the instances. Scene graph generation aims to represent the image as a semantic structural graph, where objects and relationships between them are described as nodes and edges, respectively. Some existing methods rely only on visual features to sequentially predict the relationships between objects, ignoring contextual information and making it difficult to generate high-quality scene graphs, especially for remote sensing images. Therefore, we propose a novel model for remote sensing image scene graph generation by fusing contextual information and statistical knowledge, namely RSSGG_CS. To integrate contextual information and calculate attention among all objects, the RSSGG_CS model adopts a filter module (FiM) that is based on adjusted transformer architecture. Moreover, to reduce the blindness of the model when searching semantic space, statistical knowledge of relational predicates between objects from the training dataset and the cleaned Wikipedia text is used as supervision when training the model. Experiments show that fusing contextual information and statistical knowledge allows the model to generate more complete scene graphs of remote sensing images and facilitates the semantic understanding of remote sensing images.

Hashing has been widely applied to the large-scale approximate nearest neighbor search problem owing to its high efficiency and low storage requirement. Most investigations concentrate on learning hashing methods in a centralized setting. However, in existing big data systems, data is often stored across different nodes. In some situations, data is even collected in a distributed manner. A straightforward way to solve this problem is to aggregate all the data into the fusion center to obtain the search result (aggregating method). However, this strategy is not feasible because of the prohibitive communication cost. Although a few distributed hashing methods have been proposed to reduce this cost, they only focus on designing a distributed algorithm for a specific global optimization objective without considering scalability. Moreover, existing distributed hashing methods aim at finding a distributed solution to hashing, meanwhile avoiding accuracy loss, rather than improving accuracy. To address these challenges, we propose a Scalable Distributed Hashing (SDisH) model in which most existing hashing methods can be extended to process distributed data with no changes. Furthermore, to improve accuracy, we utilize the search radius as a global variable across different nodes to achieve a global optimum search result for every iteration. In addition, a voting algorithm is presented based on the results produced by multiple iterations to further reduce search errors. Theoretical analyses of communication, computation, and accuracy demonstrate the superiority of the proposed model. Numerical simulations on three large-scale and two relatively small benchmark datasets also show that the SDisH model achieves up to 44.75% and 10.23% accuracy gains compared to the aggregating method and state-of-the-art distributed hashing methods, respectively.

Scene graph generation refers to the task of automatically mapping an image into a semantic structural graph, which requires correctly labeling each extracted object and their interaction relationships. Despite the recent success in object detection using deep learning techniques, inferring complex contextual relationships and structured graph representations from visual data remains a challenging topic. In this study, we propose a novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem. The first module is a semantic transformation module utilized to capture semantic embedded relation features, by translating visual features and linguistic features into a common semantic space. The other module is a graph self-attention module introduced to embed a joint graph representation through assigning various importance weights to neighboring nodes. Finally, accurate scene graphs are produced by the relation inference module to recognize all entities and corresponding relations. We evaluate our proposed method on the widely-adopted Visual Genome Dataset, and the results demonstrate the effectiveness and superiority of our model.

Text-to-Image Person Retrieval (TIPR) aims to retrieve person images based on natural language descriptions. Although many TIPR methods have achieved promising results, sometimes textual queries cannot accurately and comprehensively reflect the content of the image, leading to poor cross-modal alignment and overfitting to limited datasets. Moreover, the inherent modality gap between text and image further amplifies these issues, making accurate cross-modal retrieval even more challenging. To address these limitations, we propose the Generation-Enhanced Alignment (GEA) from a generative perspective. GEA contains two parallel modules: 1) Text-Guided Token Enhancement (TGTE), which introduces diffusion-generated images as intermediate semantic representations to bridge the gap between text and visual patterns. These generated images enrich the semantic representation of text and facilitate cross-modal alignment. 2) Generative Intermediate Fusion (GIF) module, which combines cross-attention between generated images, original images, and text features to generate a unified representation optimized by triplet alignment loss. We conduct extensive experiments on three public TIPR datasets, CUHK-PEDES, RSTPReid, and ICFG-PEDES, to evaluate the performance of GEA. The remarkable results justify the efficacy of our method. More implementation details and extended results are available at https://github.com/sugelamyd123/Sup-for-GEA.

Cross-view localization between Unmanned Aerial Vehicle (UAV) and satellite imagery is crucial for autonomous navigation in GPS-denied environments. However, large domain gaps, including viewpoint discrepancies, scale variations, and appearance differences — pose significant challenges. In this paper, we propose the Adaptive Multi-scale Cross-modal Fusion Network (AMCF-Net), a novel approach that effectively addresses these limitations through a shared backbone architecture and adaptive fusion mechanisms. Unlike previous dual-backbone approaches that process UAV and satellite images separately, our method employs a unified FocalNet-Tiny backbone to extract cross-modal features, followed by a Spatially-adaptive Crossmodal Feature Fusion (AMCF) module that dynamically combines multi-scale similarities using learned adaptive weights. This shared representation learning enables better cross-modal alignment and significantly reduces computational overhead. Comprehensive experiments on the UL14 benchmark demonstrate that AMCF-Net achieves state-of-the-art performance, with a Relative Distance Score (RDS) of 78.12% and meter-level accuracy of 27.25% at 3 m, 50.16% at 5 m, 84.37% at 10 m, and finally 88.51% at 20 m. Ablation studies further validate the effectiveness of the shared backbone and adaptive fusion mechanism, demonstrating significant improvements over traditional separate processing approaches.

Recent advances in vision-language cross-modal learning have substantially improved the performance of video temporal grounding. However, most existing methods directly associate global video features with sentence-level features, overlooking the fact that textual semantics usually correspond to only limited spatio-temporal regions within a video. This limitation often leads to unstable alignment in complex scenarios involving intertwined events and diverse actions. In essence, accurate video temporal grounding requires the joint modeling of fine-grained spatial semantics and heterogeneous temporal event structures. Motivated by this observation, we propose a hierarchical prototype alignment approach that models cross-modal correspondence between video and text through structured intermediate prototype representations. Specifically, the alignment process is decomposed into two complementary stages: object-phrase alignment and event-sentence alignment. In the object-phrase alignment stage, discriminative local visual regions and informative textual words are aggregated to construct object and phrase prototypes, thereby enhancing fine-grained spatial correspondence at the level of entities and localized actions. In the event-sentence alignment stage, object prototypes are further integrated along the temporal dimension to form event prototypes that represent continuous action units, enabling effective alignment with sentence-level semantics and facilitating the modeling of diverse temporal event structures. On this basis, we further directly inject cross-modal alignment information into candidate moment aggregation. This design allows candidate moment representations to emphasize query-relevant temporal regions. Extensive experiments on Charades-STA, ActivityNet Captions, and TACoS demonstrate that the proposed method outperforms existing approaches, validating the effectiveness of hierarchical prototype alignment for improving both cross-modal alignment quality and temporal grounding accuracy.

Vision-language models like CLIP demonstrate exceptional generalization capabilities but face significant adaptation challenges due to parameter scale, prompt sensitivity, and cross-modal alignment difficulties.Existing approaches primarily focus on single-modality adjustments, leading to suboptimal alignment and limited generalization.We introduce MAFMO, a plugand-play framework comprising: (1) a Harmonic Cross-Modal Adapter enabling efficient cross-modal knowledge transfer; (2) a Meta-Template Optimization module dynamically generating input-dependent templates; and (3) a Cross-Modal Knowledge Synthesis mechanism preserving critical structural relationships during adaptation.Extensive experiments across multiple fine-grained visual recognition benchmarks demonstrate MAFMO consistently improves existing methods' performance on both novel classes and harmonic mean, while maintaining robustness under various challenging conditions with minimal computational overhead.

Recently, the attention-enriched encoder-decoder framework has aroused great interest in image captioning due to its overwhelming progress. Many visual attention models directly leverage meaningful regions to generate image descriptions. However, seeking a direct transition from visual space to text is not enough to generate fine-grained captions. This paper exploits a feature-compounding approach to bring together high-level semantic concepts and visual information regarding the contextual environment fully end-to-end. Thus, we propose a stacked cross-modal feature consolidation (SCFC) attention network for image captioning in which we simultaneously consolidate cross-modal features through a novel compounding function in a multi-step reasoning fashion. Besides, we jointly employ spatial information and context-aware attributes (CAA) as the principal components in our proposed compounding function, where our CAA provides a concise context-sensitive semantic representation. To make better use of consolidated features potential, we further propose an SCFC-LSTM as the caption generator, which can leverage discriminative semantic information through the caption generation process. The experimental results indicate that our proposed SCFC can outperform various state-of-the-art image captioning benchmarks in terms of popular metrics on the MSCOCO and Flickr30K datasets.

One of the core challenges in natural language processing (NLP) lies in achieving deep semantic understanding, which goes beyond shallow semantic matching to capture the complex emotions, intentions, and contextual dependencies embedded in multimodal data. This paper proposes a novel deep semantic understanding model that introduces a hierarchical multimodal fusion mechanism and a self-supervised cross-modal alignment paradigm. First, dedicated encoders extract fine-grained features from text, vision, and audio modalities. Then, an attention-driven hierarchical fusion network dynamically integrates and filters salient information at the feature level, semantic level, and decision level, ensuring efficient cooperation across modalities. Simultaneously, this paper designs a contrastive learning-based alignment module that narrows the representational gap between semantically related multimodal pairs while pushing apart irrelevant ones in a shared semantic space, thereby implicitly learning deep semantic consistency and alleviating the “modality gap” problem. The proposed method is systematically validated on multiple public benchmark datasets as well as domain-specific multimodal datasets. Experimental results demonstrate that our approach consistently outperforms strong baselines, including CNN, BiLSTM, Transformer, and Multimodal-BERT. Specifically, the proposed method achieves 94.7% accuracy and an F1-score of 94.2% for multimodal sentiment recognition, surpassing Multimodal-BERT by approximately 3.4% in F1-score. In cross-dataset transfer tasks, the model maintains strong generalization, achieving 90.5% F1-score when transferring from Dataset-A to Dataset-B, and exhibits faster convergence with lower variance in validation accuracy. These findings not only provides a powerful solution for multimodal semantic understanding but also offers a new technical path for exploring the development of cognitive intelligence in machines.

This paper reorganized the dictionary meanings of Japanese "nomu," Korean "masida," and Chinese "he," which exhibit corresponding meanings, and based on this, examines the semantic relationships among these verbs.Specifically, the 11 meanings of Japanese "nomu" listed in the dictionary were reorganized into 10 meanings: (1) to put something in the mouth and swallow, (2) to inhale, (3) to accept, (4) to belittle, (5) to endure, (6) to take in, (7) to embezzle, (8) to hide and keep, (9) to transact on behalf of someone, and (10) to bet (especially on horse racing).The semantic scope of the Chinese "he" was reorganized from two meanings to one.Based on these semantic ranges, the semantic relationships among "nomu," "masida," and "he" were diagrammed as shown in [Fig.1].In addition, from a cognitive semantics perspective, the cognitive ranges used by native speakers of Japanese "nomu," Korean "masida," and Chinese "he" diagrammed for comparison.The results indicated that the cognitive scope of "masida" and "he" was similar.In contrast, the scope of "nomu" was considerably broader than that of "masida" in Korea or "he" in Chinese.Further, cognitive semantic models for these lexical items were presented in [Fig.4], [Fig.5 and 6], and [Fig.7 and 8], respectively.