DeeCap: Dynamic Early Exiting for Efficient Image Captioning

Comparative study of trophic and elemental characteristics of zooplankton in deep (500–3500 m) and shallow (0–200 m) layers

Unsupervised image captioning often grapples with challenges such as image–text mismatches and modality gaps, resulting in suboptimal captions. This paper introduces a semantic-enhanced cross-modal fusion model (SCFM) to address these issues. The SCFM integrates three innovative components: a text semantic enhancement network (TSE-Net) for nuanced semantic representation; contrast learning for optimizing similarity measures between text and images; and enhanced visual selection decoding (EVSD) for precise captioning. Unlike existing methods that struggle with capturing accurate semantic relationships and flexibility across scenarios, the proposed model provides a robust solution for unbiased and diverse captioning. Through experimental evaluations on the MS COCO and Flickr30k datasets, SCFM demonstrates significant improvements over the benchmark model, enhancing the CIDEr and BLEU-4 metrics by 3.6% and 3.2%, respectively. Visualization analysis further reveals the model’s superiority in increasing variability between hidden features and its potential in cross-domain and stylized image captioning. The findings not only contribute to the advancement of image captioning techniques but also open avenues for future research. Further investigations will explore SCFM’s adaptability to other multimodal tasks and refine it for more intricate image–text relationships.

Transform, contrast and tell: Coherent entity-aware multi-image captioning

Image captioning refers to automatic generation of descriptive texts according to the visual content of images. It is a technique integrating multiple disciplines including the computer vision (CV), natural language processing (NLP) and artificial intelligence. In recent years, substantial research efforts have been devoted to generate image caption with impressive progress. To summarize the recent advances in image captioning, we present a comprehensive review on image captioning, covering both traditional methods and recent deep learning-based techniques. Specifically, we first briefly review the early traditional works based on the retrieval and template. Then deep learning-based image captioning researches are focused, which is categorized into the encoder-decoder framework, attention mechanism and training strategies on the basis of model structures and training manners for a detailed introduction. After that, we summarize the publicly available datasets, evaluation metrics and those proposed for specific requirements, and then compare the state of the art methods on the MS COCO dataset. Finally, we provide some discussions on open challenges and future research directions.

Automatically describing the information of an image using properly constructed sentences is a tricky task in any language. However, it has the potential to have a significant effect by enabling visually challenged individuals to better understand their surroundings. This paper proposes an image captioning system that generates detailed captions and extracts text from an image, if any, and uses it as a part of the caption to provide a more precise description of the image. To extract the image features, the proposed model uses Convolutional Neural Networks (CNNs) followed by Long Short-Term Memory (LSTM) that generates corresponding sentences based on the learned image features. Further, using the text extraction module, the extracted text (if any) is included in the image description and the captions are presented in audio form. Publicly available benchmark datasets for image captioning like MS COCO, Flickr-8k, Flickr-30k have a variety of images, but they hardly have images that contain textual information. These datasets are not sufficient for the proposed model and this has resulted in the creation of a new image caption dataset that contains images with textual content. With the newly created dataset, comparative analysis of the experimental results is performed on the proposed model and the existing pre-trained model. The obtained experimental results show that the proposed model is equally effective as the existing one in subtitle image captioning models and provides more insights about the image by performing text extraction.

CNN-LSTM based architectures have played an important role in image captioning, but limited by the training efficiency and expression ability, researchers began to explore the CNN-Transformer based models and achieved great success. Meanwhile, almost all recent works adopt Faster R-CNN as the backbone encoder to extract region-level features from given images. However, Faster R-CNN needs a pre-training on an additional dataset, which divides the image captioning task into two stages and limits its potential applications. In this paper, we build a pure Transformer-based model, which integrates image captioning into one stage and realizes end-to-end training. Firstly, we adopt SwinTransformer to replace Faster R-CNN as the backbone encoder to extract grid-level features from given images; Then, referring to Transformer, we build a refining encoder and a decoder. The refining encoder refines the grid features by capturing the intra-relationship between them, and the decoder decodes the refined features into captions word by word. Furthermore, in order to increase the interaction between multi-modal (vision and language) features to enhance the modeling capability, we calculate the mean pooling of grid features as the global feature, then introduce it into refining encoder to refine with grid features together, and add a pre-fusion process of refined global feature and generated words in decoder. To validate the effectiveness of our proposed model, we conduct experiments on MSCOCO dataset. The experimental results compared to existing published works demonstrate that our model achieves new state-of-the-art performances of 138.2% (single model) and 141.0% (ensemble of 4 models) CIDEr scores on 'Karpathy' offline test split and 136.0% (c5) and 138.3% (c40) CIDEr scores on the official online test server. Trained models and source code will be released.

Self-attention mechanism, which has been successfully applied to current encoder-decoder framework of image captioning, is used to enhance the feature representation in the image encoder and capture the most relevant information for the language decoder. However, most existing methods will assign attention weights to all candidate vectors, which implicitly hypothesizes that all vectors are relevant. Moreover, current self-attention mechanisms ignore the intra-object attention distribution, and only consider the inter-object relationships. In this paper, we propose a Multi-Gate Attention (MGA) block, which expands the traditional self-attention by equipping with additional Attention Weight Gate (AWG) module and Self-Gated (SG) module. The former constrains the attention weights to be assigned to the most contributive objects. The latter is adopted to consider the intra-object attention distribution and eliminate the irrelevant information in object feature vector. Furthermore, most current image captioning methods apply the original transformer designed for natural language processing task, to refine image features directly. Therefore, we propose a pre-layernorm transformer to simplify the transformer architecture and make it more efficient for image feature enhancement. By integrating MGA block with pre-layernorm transformer architecture into the image encoder and AWG module into the language decoder, we present a novel Multi-Gate Attention Network (MGAN). The experiments on MS COCO dataset indicate that the MGAN outperforms most of the state-of-the-art, and further experiments on other methods combined with MGA blocks demonstrate the generalizability of our proposal.

Recently, many excellent algorithms have made great progress in object detection, but there are also problems in these algorithms’ performance on targets of different sizes, and in particular in small object detection. Aiming at the problem of insufficient feature representation by the feature extractor, in this paper we propose a lightweight algorithm to improve feature extraction. The algorithm includes three modules. First, considering that the shallow features in feature extraction contain much background noise, in this paper we design a multi-level feedback propagation model based on a Gaussian high-pass filter. The shallow layers are enhanced using the filter and then back-propagated to add the upper shallow layer features and obtain new shallow layer features. This process is performed on the newly generated shallow layer for n iterations, which is beneficial for enhancing targets in the foreground area and suppressing background noise. Second, we form a stacked dilated convolution module with different dilation rates to cover the entire deep feature layer densely, which enlarges the receptive field and enriches the contextual information. Finally, we build a multi-scale fusion module to fuse the above-mentioned enhanced shallow and deep features to obtain output features with powerful representational ability for detection tasks. In addition, the model is easily embedded into existing approaches to enhance their performance. We build the model on the VGG-16 and ResNet-50 backbones and successfully applied it on Darknet-19 and Darknet-53 to verify its effectiveness and stability. The experiments on the COCO dataset prove that the proposed algorithm outperforms the state-of-art methods, with a mean average precision improvement reaching 2% on average. The effect is remarkable on small targets and complex backgrounds. Furthermore, it does not affect the detection speed significantly, so real time detection requirements can still be met.

VD-SAN: Visual-Densely Semantic Attention Network for Image Caption Generation

Purpose: This paper investigates the integration of multimodal generative artificial intelligence (GenAI) frameworks for enhancing image captioning and document classification, focusingparticularly on applications in healthcare. De-sign/Methodology/Approach: A systematic analysis of state-of-the-art architectures Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer-based models is conducted. We evaluate their capacity to process heterogeneous data (text and images), with experiments carried out on benchmark datasets including Reuters-21578, 20 Newsgroups, MS-COCO, Flickr30k, FUNSD, CORD, and healthcare datasets such as chest X-rays. Findings: Experimental results indicate that CNN and Hierarchical Attention Networks achieve high document classification accu-racy (>92%), while transformer-based captioning systems outperform traditional models with BLEU (35%), METEOR (28%), CIDEr (105%), and SPICE (22%). Instruction-tuned Vision-Language Models (VLMs) such as InstructBLIP and LLa-VA-1.5 further enhance zero-shot captioning and multimodal reasoning. In healthcare applications, multimodal integration improves diagnostic imaging interpretation, clinical note classification, and automated report generation. Practical Implica-tions: The framework demonstrates potential in domains requiring context-rich outputs, including medical imaging, pa-tient data analysis, and cross-domain information retrieval. Originality/Value: Unlike previous studies, this work offers a consolidated evaluation of multimodal generative AI through comparative experimentation, integration of recent models (OCR-free document AI, RAG, SAM-based grounding), and ethical considerations, establishing a roadmap for future re-search.

Visual attention has been successfully applied in image captioning to selectively incorporate the most relevant areas to the language generation procedure. However, the attention in current image captioning methods is only guided by the hidden state of language model, e.g. LSTM (Long-Short Term Memory), indirectly and implicitly, and thus the attended areas are weakly relevant at different time steps. Besides the spatial relationship of attention areas, the temporal relationship in attention is crucial for image captioning according to the attention transmission mechanism of human vision. In this paper, we propose a new spatio-temporal memory attention (STMA) model to learn the spatio-temporal relationship in attention for image captioning. The STMA introduces the memory mechanism to the attention model through a tailored LSTM, where the new cell is used to memorize and propagate the attention information, and the output gate is used to generate attention weights. The attention in STMA transmits with memory adaptively and dependently, which builds strong temporal connections of attentions and learns the spatio-temporal relationship of attended areas simultaneously. Besides, the proposed STMA is flexible to combine with attention-based image captioning frameworks. Experiments on MS COCO dataset demonstrate the superiority of the proposed STMA model in exploring the spatio-temporal relationship in attention and improving the current attention-based image captioning.

Rainfed lowland rice fields are characterized by soil moisture fluctuations (SMF) and the presence of hardpan that impedes deep rooting and thus limits water extraction from deep soil layer during the periods of drought. In this study, we used rootboxes with three layers; shallow layer, artificial hardpan, and deep and wet layer below the hardpan, to evaluate differences in the plasticity of nodal roots elongation through the hardpan and promote root branching below the hardpan in response to SMF among four rice varieties; Sasanishiki, Habataki, Nipponbare, and Kasalath. Experiments were conducted during the summer and autumn seasons. Plasticity was computed as the difference in root traits within each variety between the SMF and continuously well-watered treatments. In both experiments, Habataki consistently tended to exhibit higher root plasticity than the other three varieties by increasing number of nodal roots that penetrated the hardpan during rewatering period in SMF, when the soil moisture increased and penetration resistance decreased. This root plasticity then contributed to greater water use at the deeper soil during the subsequent drought period and overall shoot dry matter production. Habataki had significantly higher δ13C value in roots at deep layer than roots at the shallow and hardpan layers under SMF, which may indicate that these were relatively newly grown roots as a consequence of root plasticity. This study also indicates that CSSLs derived from Sasanishiki and Habataki varieties may be suitable for the analysis of QTLs associated with root plasticity expression in rainfed lowland with hardpan and experiencing SMF.

Image Captioning is the process of generating textual descriptions of an image, which need to be syntactically and semantically correct. This paper extensively surveys very early literature that includes the advent of Artificial Intelligence, the Machine Learning pathway, the early Deep Learning and the current Deep Learning methodology for Image Captioning. This survey paper aims to develop a system to predict captions for the given images with a higher accuracy by combining the results of different Deep Learning Techniques. This model based on a neural network consists of a vision CNN followed by the language generator RNN. It generates complete sentence in natural language from an input image. The state of art is achieved by comparing three different encoder –decoder models. By comparing three models, the blue score of CNN-LSTM Model with Flikr 8k dataset is 0.44, CNN-LSTM Word Embedding with Flikr 8k dataset is 0.68 and CNN –GRU model Visual Attention with MSCOCO Dataset is 0.86.

Image captioning aims at understanding various semantic concepts (e.g., objects and relationships) from an image and integrating them in a sentence-level description. Hence, it is necessary to learn the interaction among these concepts. If we define the context of the interaction to be involved in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">subject-predicate-object</i> triplet, most current methods only focus on the single triplet for the first-order interaction to generate sentences. Intuitively, we humans are able to perceive the high-order interaction among concepts from two or more triplets to describe an image. For example, when we see the triplets <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">man-cutting-sandwich</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">man-with-knife</i> , it is natural to integrate and predict the sentence <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">man cutting sandwich with knife</i> . This depends on the high-order interaction between <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cutting</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">knife</i> in different triplets. Therefore, exploiting high-order interaction is expected to benefit image captioning and focus on reasoning. In this paper, we introduce the novel high-order interaction learning method over detected objects and relationships for image captioning under the umbrella of the encoder-decoder framework. We first extract a set of object and relationship features in an image. During the encoding stage, the interactive refining network is proposed to learn high-order representations by modeling intra- and inter-object feature interaction in the self-attention fashion. During the decoding stage, the interactive fusion network is proposed to integrate object and relationship information by strengthening their high-order interaction based on language context for sentence generation. In this way, we learn the object-relationship dependencies in different stages, which can provide abundant cues for both visual understanding and caption generation. Extensive experiments show that the proposed method can achieve competitive performances against the state-of-the-art methods on MSCOCO dataset. Additional ablation studies further validate its effectiveness.

Current state-of-the-art image captioning models generate captions in a single language, requiring a combination of multiple language specific models to build a multilingual image captioning system. However, as the number of supported languages increases, it leads to the parameters of the multilingual image captioning system grow linearly. To tackle this issue, we propose a single Compact Interactive TransformEr (CITE) model, which can describe an image in multiple languages simultaneously, making the captioning system more compact. Specifically, based on the standard Transformer model, we share the encoder and decoder backbone parameters and replace the self-attention sub-layer in the decoder with the interactive attention sub-layer. In addition, we extend the traditional monolingual reinforcement learning mechanism to a multilingual version to promote better description generation. Due to the wide use of Chinese and English, we evaluate the performance of our CITE model by simultaneously generating English and Chinese captions. We expand the image captions of the whole MSCOCO dataset and release a COCO-EN-CN dataset. Extensive experiments on the COCO-EN-CN dataset show that our single CITE model with more parameter-efficient can maintain the competitive performance or even better than the monolingual captioning models.