Image Caption Generation Using Neural Network Models and LSTM Hierarchical Structure

Image Captioning, the process of generating a textual description of an image, has emerged as a hot research due to its practical importance in many domains. It is a challenging task as it uses both Natural Language Processing and Computer Vision related fields to generate the captions. Despite the fact that the literature has reported notable image captioning methodologies, they still lag in accomplishing the substantial performance level for diverse datasets. This paper proposes an image caption generating mechanism based on Optimized Bidirectional Long Short-Term Memory (B-LSTM) model. We propose a variant of Moth Flame Optimization (PMFO), termed here as Proposed Moth Flame Optimization (PMFO), which has logarithmic spiral update based on correlation. The performance of the proposed model is demonstrated on benchmark datasets like Flicker 8 k, Flicker30k, VizWik and COCO datasets using renowned metrics such as CIDEr, BLEU, SPICE and ROUGH. The performance analysis proves that the B-LSTM achieves better performance on caption generation than state-of-the-art methods.

Recent advancements in technology have made available a variety of image capturing devices, ranging from handheld mobiles to space-grade rovers. This has generated a tremendous visual data, which has made a necessity to organize and understand this visual data. Thus, there is a need to caption thousands of such images. This has resulted in tremendous research in computer vision and deep learning. Inspired by such recent works, we present an image caption generating system that uses convolutional neural network (CNN) for extracting the feature embedding of an image and feed that as an input to long short-term memory cells that generates a caption. We are using two pre-trained CNN models on ImageNet, VGG16 and ResNet-101. Both the models were tested and compared on Flickr8K dataset. Experimental results on this dataset demonstrate that the proposed architecture is as efficient as the state-of-the-art multi-label classification models. Experimental results on public benchmark dataset demonstrate that the proposed architecture performs as efficiently as the state-of-the-art image captioning model.

The field of Artificial Intelligence had made a huge change in all sectors of computer technology. It had given a comfort zone and user friendly environment for human beings. Similarly the most trending topic also made a lots of improvement in Image captioning. Image captioning is most widely used technology in computer field. In this paper we will be implementing generation of image captions using neural network models and understand what the role of this models is in caption generation. The models used for implementation are CNN, RNN with LSTM. In this paper the dataset used is Flickr 8k dataset which consist images of different varieties and different activities performed by human beings. The main aim is to extract all the features from the given input image and establish an appropriate relationship from the extracted features and tried to get the most accurate captions for given input images.

Image captioning, which aims to automatically generate a sentence description for an image, has attracted much research attention in cognitive computing. The task is rather challenging, since it requires cognitively combining the techniques from both computer vision and natural language processing domains. Existing CNN-RNN framework-based methods suffer from two main problems: in the training phase, all the words of captions are treated equally without considering the importance of different words; in the caption generation phase, the semantic objects or scenes might be misrecognized. In our paper, we propose a method based on the encoder-decoder framework, named Reference based Long Short Term Memory (R-LSTM), aiming to lead the model to generate a more descriptive sentence for the given image by introducing reference information. Specifically, we assign different weights to the words according to the correlation between words and images during the training phase. We additionally maximize the consensus score between the captions generated by the captioning model and the reference information from the neighboring images of the target image, which can reduce the misrecognition problem. We have conducted extensive experiments and comparisons on the benchmark datasets MS COCO and Flickr30k. The results show that the proposed approach can outperform the state-of-the-art approaches on all metrics, especially achieving a 10.37% improvement in terms of CIDEr on MS COCO. By analyzing the quality of the generated captions, we come to a conclusion that through the introduction of reference information, our model can learn the key information of images and generate more trivial and relevant words for images.

The image caption generation algorithm necessitates the expression of image content using accurate natural language. Given the existing encoder-decoder algorithm structure, the decoder solely generates words one by one in a front-to-back order and is unable to analyze integral contextual information. This paper employs a Bi-LSTM (Bi-directional Long Short-Term Memory) structure, which not only draws on past information but also captures subsequent information, resulting in the prediction of image content subject to the context clues. The visual information is respectively fed into the F-LSTM decoder (forward LSTM decoder) and B-LSTM decoder (backward LSTM decoder) to extract semantic information, along with complementing semantic output. Specifically, the subsidiary attention mechanism S-Att acts between F-LSTM and B-LSTM, while the semantic information of B-LSTM and F-LSTM is extracted using the attention mechanism. Meanwhile, the semantic interaction is extracted pursuant to the similarity while aligning the hidden states, resulting in the output of the fused semantic information. We adopt a Bi-LSTM-s model capable of extracting contextual information and realizing finer-grained image captioning effectively. In the end, our model improved by 9.7% on the basis of the original LSTM. In addition, our model effectively solves the problem of inconsistent semantic information in the forward and backward direction of the simultaneous order, and gets a score of 37.5 on BLEU-4. The superiority of this approach is experimentally demonstrated on the MSCOCO dataset.

In recent years, with the rapid development of artificial intelligence, image caption has gradually attracted the attention of many researchers in the field of artificial intelligence and has become an interesting and arduous task. Image caption, automatically generating natural language descriptions according to the content observed in an image, is an important part of scene understanding, which combines the knowledge of computer vision and natural language processing. The application of image caption is extensive and significant, for example, the realization of human-computer interaction. This paper summarizes the related methods and focuses on the attention mechanism, which plays an important role in computer vision and is recently widely used in image caption generation tasks. Furthermore, the advantages and the shortcomings of these methods are discussed, providing the commonly used datasets and evaluation criteria in this field. Finally, this paper highlights some open challenges in the image caption task.

Enhancing image caption generation through context-aware attention mechanism

Image caption generation is a fundamental task to build a bridge between image and its description in text, which is drawing increasing interest in artificial intelligence. Images and textual sentences are viewed as two different carriers of information, which are symmetric and unified in the same content of visual scene. The existing image captioning methods rarely consider generating a final description sentence in a coarse-grained to fine-grained way, which is how humans understand the surrounding scenes; and the generated sentence sometimes only describes coarse-grained image content. Therefore, we propose a coarse-to-fine-grained hierarchical generation method for image captioning, named SDA-CFGHG, to address the two problems above. The core of our SDA-CFGHG method is a sequential dual attention that is used to fuse different grained visual information with sequential means. The advantage of our SDA-CFGHG method is that it can achieve image captioning in a coarse-to-fine-grained way and the generated textual sentence can capture details of the raw image to some degree. Moreover, we validate the impressive performance of our method on benchmark datasets—MS COCO, Flickr—with several popular evaluation metrics—CIDEr, SPICE, METEOR, ROUGE-L, and BLEU.

Automatically describing contents of an image using natural language has drawn much attention because it not only integrates computer vision and natural language processing but also has practical applications. Using an end-to-end approach, we propose a bidirectional semantic attention-based guiding of long short-term memory (Bag-LSTM) model for image captioning. The proposed model consciously refines image features from previously generated text. By fine-tuning the parameters of convolution neural networks, Bag-LSTM obtains more text-related image features via feedback propagation than other models. As opposed to existing guidance-LSTM methods which directly add image features into each unit of an LSTM block, our fine-tuned model dynamically leverages more text-conditional image features, acquired by the semantic attention mechanism, as guidance information. Moreover, we exploit bidirectional gLSTM as the caption generator, which is capable of learning long term relations between visual features and semantic information by making use of both historical and future contextual information. In addition, variations of the Bag-LSTM model are proposed in an effort to sufficiently describe high-level visual-language interactions. Experiments on the Flickr8k and MSCOCO benchmark datasets demonstrate the effectiveness of the model, as compared with the baseline algorithms, such as it is 51.2% higher than BRNN on CIDEr metric.

Recently, soft attention mechanisms have been successfully used in a wide variety of applications such as the generation of image captions, text translation, etc. This mechanism attempts to mimic the visual cortex of a human brain by not analyzing all the objects in a scene equally, but by looking for clues (or salient features) which might give a more compact representation of the environment. In doing so, the human brain can process information more quickly and without overloading. Having learned this lesson, in this paper, we try to make a bridge from the visual to the audio scene classification problem, namely the classification of heart sound signals. To do so, a novel approach merging soft attention mechanisms and recurrent neural nets is proposed. Using the proposed methodology, the algorithm can successfully learn automatically significant audio segments when detecting and classifying abnormal heart sound signals, both improving these classification results and somehow creating a simple justification for them.

Automatic creation of image descriptions, i.e. captioning of images, is an important topic in artificial intelligence (AI) that bridges the gap between computer vision (CV) and natural language processing (NLP). Currently, neural networks are becoming increasingly popular in captioning images and researchers are looking for more efficient models for CV and sequence-sequence systems. This study focuses on a new image caption generation model that is divided into two stages. Initially, low-level features, such as contrast, sharpness, color and their high-level counterparts, such as motion and facial impact score, are extracted. Then, an optimized convolutional neural network (CNN) is harnessed to generate the captions from images. To enhance the accuracy of the process, the weights of CNN are optimally tuned via spider monkey optimization with sine chaotic map evaluation (SMO-SCME). The development of the proposed method is evaluated with a diversity of metrics.

In this research, we introduce a groundbreaking methodology that leverages deep learning techniques to revolutionize the process of generating descriptive captions and hashtags for images, effectively bridging the gap between computer vision and natural language understanding. Traditional approaches to image caption generation have often relied on rudimentary techniques such as handcrafted features and rule-based systems, which inherently struggle to capture the intricate semantics of images and adapt to diverse datasets. Recognizing these limitations, our novel framework integrates state-of-the-art convolutional neural networks (CNNs) for precise image feature extraction and recurrent neural networks (RNNs), specifically employing the ResNet-50 architecture, for seamless sequence generation. Furthermore, in addition to traditional evaluation metrics such as BLEU scores and human assessments, our system, DeepLens, introduces a groundbreaking feature: automatic hashtag generation. By meticulously analyzing both the content and context of images, DeepLens autonomously generates hashtags that enrich social media content sharing and engagement, presenting a novel paradigm in the realm of image captioning. In addition to its advancements in captioning accuracy and user experience enhancement, DeepLens offers scalability and adaptability to various domains. Its robust architecture allows for seamless integration with different datasets and environments, making it versatile for a wide range of applications. Through this comprehensive approach, we aim to not only enhance the accuracy and relevance of generated captions but also elevate the overall user experience in navigating and interacting with visual content across various platforms and applications.

The automatic narration of a natural scene is an important trait in artificial intelligence that unites computer vision and natural language processing. Caption generation is a challenging task in scene understanding. Most of the state-of-the-art methods are using deep convolutional neural network models to extract visual features of the entire image, based on which the parallel structures between images and sentences are exploited using recurrent neural networks for image captioning. However, in such models, only visual features are exploited for caption generation. This work investigated that fusion of text available in an image can give more fined-grained captioning of a scene. In this paper, we have proposed a model which incorporates a deep convolutional neural network and long short-term memory to boost the accuracy of image captioning by fusing text feature available in an image with the visual features extracted in state-of-the-art methods. We have validated the effectiveness of the proposed model on the benchmark datasets (Flickr8k and Flickr30k). The experimental outcomes illustrate that the proposed model outperformed the state-of-the-art methods for image captioning.

Generating a novel and descriptive caption of an image is drawing increasing interests in computer vision, natural language processing, and multimedia communities. In this work, we propose an end-to-end trainable deep bidirectional LSTM (Bi-LSTM (Long Short-Term Memory)) model to address the problem. By combining a deep convolutional neural network (CNN) and two separate LSTM networks, our model is capable of learning long-term visual-language interactions by making use of history and future context information at high-level semantic space. We also explore deep multimodal bidirectional models, in which we increase the depth of nonlinearity transition in different ways to learn hierarchical visual-language embeddings. Data augmentation techniques such as multi-crop, multi-scale, and vertical mirror are proposed to prevent overfitting in training deep models. To understand how our models “translate” image to sentence, we visualize and qualitatively analyze the evolution of Bi-LSTM internal states over time. The effectiveness and generality of proposed models are evaluated on four benchmark datasets: Flickr8K, Flickr30K, MSCOCO, and Pascal1K datasets. We demonstrate that Bi-LSTM models achieve highly competitive performance on both caption generation and image-sentence retrieval even without integrating an additional mechanism (e.g., object detection, attention model). Our experiments also prove that multi-task learning is beneficial to increase model generality and gain performance. We also demonstrate the performance of transfer learning of the Bi-LSTM model significantly outperforms previous methods on the Pascal1K dataset.

This project presents an image caption generator that converts images into descriptive speech to assist visually impaired users. It integrates a VGG16-based CNN for feature extraction and an attention-equipped LSTM for caption generation. Beam and greedy search improve caption quality, while pyttsx3 enables real-time audio output. Trained over 40 epochs, the model achieved a training loss of 0.4941 and validation loss of 12.23, showing potential to enhance accessibility and environmental awareness.