Transform, contrast and tell: Coherent entity-aware multi-image captioning

Image captioning extract image features and automatically describe the content of an image in words. Recently image captioning has broken through the application of natural images and is widely used in the arts. It can be applied to art retrieval and management, and it can also automatically provide artistic introductions for the visually impaired. This paper reviews related research in image captioning of artworks, and divides image captioning into three types, including template-based approach, retrieval-based approach, and generative approach. Furthermore, mainstream generative approaches include Encoder-decoder, Transformer, New generation framework, etc. Finally, this paper summarizes the evaluation metrics for image captioning, and looks forward to the application and future development of art image captioning.

Image captioning has attracted more and more attention in the field of artificial intelligence. It is a challenging task that involves natural language description generation by automatic visual data analysis. This involves understanding the visual contents of the image and then generating natural language sentences describing the image. A number of techniques have been proposed to solve this problem. This chapter presents a survey on deep learning-based image captioning techniques and the recent advances in this field. An exhaustive explanation of the evaluation metrics is also presented. The focus of this chapter is attention-guided methods since they have shown promising results for image caption generation. The existing evaluation methods are discussed in detail, along with examples to elaborate the mechanism behind them. Furthermore, the most used benchmark datasets – Flickr8k, Flick30k, and MS-COCO – are also described. Image captioning has applications in many areas. In this chapter, the application of image captioning on industrial images is presented.

Image captioning refers to automatic generation of descriptive texts according to the visual content of images. It is a technique integrating multiple disciplines including the computer vision (CV), natural language processing (NLP) and artificial intelligence. In recent years, substantial research efforts have been devoted to generate image caption with impressive progress. To summarize the recent advances in image captioning, we present a comprehensive review on image captioning, covering both traditional methods and recent deep learning-based techniques. Specifically, we first briefly review the early traditional works based on the retrieval and template. Then deep learning-based image captioning researches are focused, which is categorized into the encoder-decoder framework, attention mechanism and training strategies on the basis of model structures and training manners for a detailed introduction. After that, we summarize the publicly available datasets, evaluation metrics and those proposed for specific requirements, and then compare the state of the art methods on the MS COCO dataset. Finally, we provide some discussions on open challenges and future research directions.

Image caption generation is a fundamental task to build a bridge between image and its description in text, which is drawing increasing interest in artificial intelligence. Images and textual sentences are viewed as two different carriers of information, which are symmetric and unified in the same content of visual scene. The existing image captioning methods rarely consider generating a final description sentence in a coarse-grained to fine-grained way, which is how humans understand the surrounding scenes; and the generated sentence sometimes only describes coarse-grained image content. Therefore, we propose a coarse-to-fine-grained hierarchical generation method for image captioning, named SDA-CFGHG, to address the two problems above. The core of our SDA-CFGHG method is a sequential dual attention that is used to fuse different grained visual information with sequential means. The advantage of our SDA-CFGHG method is that it can achieve image captioning in a coarse-to-fine-grained way and the generated textual sentence can capture details of the raw image to some degree. Moreover, we validate the impressive performance of our method on benchmark datasets—MS COCO, Flickr—with several popular evaluation metrics—CIDEr, SPICE, METEOR, ROUGE-L, and BLEU.

Enhancing image caption generation through context-aware attention mechanism

Both accuracy and efficiency are crucial for image captioning in real-world scenarios. Although Transformer-based models have gained significant improved captioning performance, their computational cost is very high. A feasible way to reduce the time complexity is to exit the prediction early in internal decoding layers without passing the entire model. However, it is not straightforward to devise early exiting into image captioning due to the following issues. On one hand, the representation in shallow layers lacks high-level semantic and sufficient cross-modal fusion information for accurate prediction. On the other hand, the exiting decisions made by internal classifiers are unreliable sometimes. To solve these issues, we propose DeeCap framework for efficient image captioning, which dynamically selects proper-sized decoding layers from a global perspective to exit early. The key to successful early exiting lies in the specially designed imitation learning mechanism, which predicts the deep layer activation with shallow layer features. By deliberately merging the imitation learning into the whole image captioning architecture, the imitated deep layer representation can mitigate the loss brought by the missing of actual deep layers when early exiting is undertaken, resulting in significant reduction in calculation cost with small sacrifice of accuracy. Experiments on the MS COCO and Flickr30k datasets demonstrate the DeeCap can achieve competitive performances with 4× speed-up. Code is available at: https://github.com/feizc/DeeCap.

In this research, we introduce a groundbreaking methodology that leverages deep learning techniques to revolutionize the process of generating descriptive captions and hashtags for images, effectively bridging the gap between computer vision and natural language understanding. Traditional approaches to image caption generation have often relied on rudimentary techniques such as handcrafted features and rule-based systems, which inherently struggle to capture the intricate semantics of images and adapt to diverse datasets. Recognizing these limitations, our novel framework integrates state-of-the-art convolutional neural networks (CNNs) for precise image feature extraction and recurrent neural networks (RNNs), specifically employing the ResNet-50 architecture, for seamless sequence generation. Furthermore, in addition to traditional evaluation metrics such as BLEU scores and human assessments, our system, DeepLens, introduces a groundbreaking feature: automatic hashtag generation. By meticulously analyzing both the content and context of images, DeepLens autonomously generates hashtags that enrich social media content sharing and engagement, presenting a novel paradigm in the realm of image captioning. In addition to its advancements in captioning accuracy and user experience enhancement, DeepLens offers scalability and adaptability to various domains. Its robust architecture allows for seamless integration with different datasets and environments, making it versatile for a wide range of applications. Through this comprehensive approach, we aim to not only enhance the accuracy and relevance of generated captions but also elevate the overall user experience in navigating and interacting with visual content across various platforms and applications.

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g., whether an image generated from "a car driving down the street" contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

A survey on automatic image caption generation

Image captioning is the process of producing the sentence description from an image. It consists of two approaches: image based and language based. Extracting the features of an input image is image-based model. The feature and objects obtained from image-based model are translated into a natural sentence by language-based model. Image captioning is a challenging task as it has to determine the objects in the image and it also has to capture and express their attributes and relationships in natural languages. Deep learning-based technique paves the way for handling the challenges and complexities of image captioning. The goal is to present a literature review of machine learning and deep learning-based image captioning techniques and discusses the performances, strengths, and weaknesses. This paper also highlights the comparison of deep learning-based techniques along with image and language model with respect to the evaluation metrics like BLUE-1, BLUE-2 and METEOR on Flickr8k and MSCOCO datasets.KeywordsImage captioningDeep learningSupervised learningUnsupervised learningReinforcement learningEncoder–decoderAttentionSemantic conceptNovel object basedCNNLSTMRLSTMRNN

Many vision-language tasks can be reduced to the problem of sequence prediction for natural language output. In particular, recent advances in image captioning use deep reinforcement learning (RL) to alleviate the "exposure bias" during training: ground-truth subsequence is exposed in every step prediction, which introduces bias in test when only predicted subsequence is seen. However, existing RL-based image captioning methods only focus on the language policy while not the visual policy (e.g., visual attention), and thus fail to capture the visual context that are crucial for compositional reasoning such as visual relationships (e.g., "man riding horse") and comparisons (e.g., "smaller cat"). To fill the gap, we propose a Context-Aware Visual Policy network (CAVP) for sequence-level image captioning. At every time step, CAVP explicitly accounts for the previous visual attentions as the context, and then decides whether the context is helpful for the current word generation given the current visual attention. Compared against traditional visual attention that only fixes a single image region at every step, CAVP can attend to complex visual compositions over time. The whole image captioning model --- CAVP and its subsequent language policy network --- can be efficiently optimized end-to-end by using an actor-critic policy gradient method with respect to any caption evaluation metric. We demonstrate the effectiveness of CAVP by state-of-the-art performances on MS-COCO offline split and online server, using various metrics and sensible visualizations of qualitative visual context. The code is available at https://github.com/daqingliu/CAVP

Image caption generation becomes a raising topic in computer vision and artificial intelligence. In order to solve the problem of stiff description, we intend to extract richer features using convolutional neural network (CNN). A neural and probabilistic framework has been proposed consequently which combines CNN with a special form of recurrent neural network (RNN) to produce an end-to-end image captioning. We use a model that takes advantage of word to vector to encode the variable length input into a fixed dimensional vector. Considering the description of the object in an image is not specific enough, we introduce an attention mechanism through visualization to show how the model is able to fix its gaze on salient objects. We validate our model on three benchmark datasets and get great performance by using standard evaluation metrics.

In recent years, with the rapid development of artificial intelligence, image caption has gradually attracted the attention of many researchers in the field of artificial intelligence and has become an interesting and arduous task. Image caption, automatically generating natural language descriptions according to the content observed in an image, is an important part of scene understanding, which combines the knowledge of computer vision and natural language processing. The application of image caption is extensive and significant, for example, the realization of human-computer interaction. This paper summarizes the related methods and focuses on the attention mechanism, which plays an important role in computer vision and is recently widely used in image caption generation tasks. Furthermore, the advantages and the shortcomings of these methods are discussed, providing the commonly used datasets and evaluation criteria in this field. Finally, this paper highlights some open challenges in the image caption task.

CNN-LSTM based architectures have played an important role in image captioning, but limited by the training efficiency and expression ability, researchers began to explore the CNN-Transformer based models and achieved great success. Meanwhile, almost all recent works adopt Faster R-CNN as the backbone encoder to extract region-level features from given images. However, Faster R-CNN needs a pre-training on an additional dataset, which divides the image captioning task into two stages and limits its potential applications. In this paper, we build a pure Transformer-based model, which integrates image captioning into one stage and realizes end-to-end training. Firstly, we adopt SwinTransformer to replace Faster R-CNN as the backbone encoder to extract grid-level features from given images; Then, referring to Transformer, we build a refining encoder and a decoder. The refining encoder refines the grid features by capturing the intra-relationship between them, and the decoder decodes the refined features into captions word by word. Furthermore, in order to increase the interaction between multi-modal (vision and language) features to enhance the modeling capability, we calculate the mean pooling of grid features as the global feature, then introduce it into refining encoder to refine with grid features together, and add a pre-fusion process of refined global feature and generated words in decoder. To validate the effectiveness of our proposed model, we conduct experiments on MSCOCO dataset. The experimental results compared to existing published works demonstrate that our model achieves new state-of-the-art performances of 138.2% (single model) and 141.0% (ensemble of 4 models) CIDEr scores on 'Karpathy' offline test split and 136.0% (c5) and 138.3% (c40) CIDEr scores on the official online test server. Trained models and source code will be released.

Large language models can provide detailed contextual understanding of a scene beyond what computer vision alone can provide, which has implications for robotics and embodied intelligence. In this study, we developed a novel multi-modal vision-language system for egocentric visual perception, with an initial focus on real-world walking environments. We trained a number of state-of-the-art transformer-based vision-language models that use causal language modelling on our custom dataset of 43,055 image-text pairs for few-shot image captioning. We designed a new speech synthesis model and a user interface to convert the generated image captions into speech for audio feedback to users. Our system also uniquely allows for feedforward user prompts to personalize the generated image captions by incorporating human cognition in the decision making. Our system is able to generate detailed captions with an average length of 10 words while achieving a high ROUGE-L score of 43.9% and a low word error rate of 28.1% with an end-to-end processing time of 2.2 seconds. Overall, our new multimodal vision-language system can generate accurate and detailed descriptions of scenes, which can be further augmented by user prompts. This innovative feature allows our image captions to be personalized to the individual needs and preferences of the user, thus optimizing the closed-loop interactions between the human and generative AI models for understanding and navigating of real-world environments.