Parallel-fusion LSTM with synchronous semantic and visual information for image captioning

The image caption generation algorithm necessitates the expression of image content using accurate natural language. Given the existing encoder-decoder algorithm structure, the decoder solely generates words one by one in a front-to-back order and is unable to analyze integral contextual information. This paper employs a Bi-LSTM (Bi-directional Long Short-Term Memory) structure, which not only draws on past information but also captures subsequent information, resulting in the prediction of image content subject to the context clues. The visual information is respectively fed into the F-LSTM decoder (forward LSTM decoder) and B-LSTM decoder (backward LSTM decoder) to extract semantic information, along with complementing semantic output. Specifically, the subsidiary attention mechanism S-Att acts between F-LSTM and B-LSTM, while the semantic information of B-LSTM and F-LSTM is extracted using the attention mechanism. Meanwhile, the semantic interaction is extracted pursuant to the similarity while aligning the hidden states, resulting in the output of the fused semantic information. We adopt a Bi-LSTM-s model capable of extracting contextual information and realizing finer-grained image captioning effectively. In the end, our model improved by 9.7% on the basis of the original LSTM. In addition, our model effectively solves the problem of inconsistent semantic information in the forward and backward direction of the simultaneous order, and gets a score of 37.5 on BLEU-4. The superiority of this approach is experimentally demonstrated on the MSCOCO dataset.

Dense semantic embedding network for image captioning

The traditional image caption algorithms output the image description text based on the image features. However, the predicted results often lack crucial semantic and spatial information. To solve the above problems, this paper proposes an image caption model based on multimodal data integration, which improves the ability of the image caption model to capture critical semantic information by adding mouse trace and object bounding box as additional information sources of the model. Firstly, the mouse traces are segmented, transformed into bounding boxes, and aligned with a text word mark. Secondly, we utilize the object detection network to learn the image features and predict the bounding box used to supplement image features. Finally, the multi-head attention mechanism is used to encode, integrate, and decode the image, caption, and mouse trace, and predict the image caption results. To verify the effectiveness of this method, the proposed method was tested on ADE20k, Flickr30k, and MSCOCO data sets based on localized narrative annotation achieving CIDEr scores of 1.280, 1.983, and 1.355. Compared with the existing methods, the proposed method has obvious advantages.

Both accuracy and efficiency are crucial for image captioning in real-world scenarios. Although Transformer-based models have gained significant improved captioning performance, their computational cost is very high. A feasible way to reduce the time complexity is to exit the prediction early in internal decoding layers without passing the entire model. However, it is not straightforward to devise early exiting into image captioning due to the following issues. On one hand, the representation in shallow layers lacks high-level semantic and sufficient cross-modal fusion information for accurate prediction. On the other hand, the exiting decisions made by internal classifiers are unreliable sometimes. To solve these issues, we propose DeeCap framework for efficient image captioning, which dynamically selects proper-sized decoding layers from a global perspective to exit early. The key to successful early exiting lies in the specially designed imitation learning mechanism, which predicts the deep layer activation with shallow layer features. By deliberately merging the imitation learning into the whole image captioning architecture, the imitated deep layer representation can mitigate the loss brought by the missing of actual deep layers when early exiting is undertaken, resulting in significant reduction in calculation cost with small sacrifice of accuracy. Experiments on the MS COCO and Flickr30k datasets demonstrate the DeeCap can achieve competitive performances with 4× speed-up. Code is available at: https://github.com/feizc/DeeCap.

We receive different impressions from different visual images or odors, but it was not clear what kinds of impression the combination of visual images and odors elicits. Thus, we conducted experiments where we compared affective impressions when only olfactory information, only visual information, and combinations of olfactory and visual information were presented. In this experiment, the subjects were asked to rate the impression in ‘mild-sharp’ and ‘clear-dense’ dimensions using a visual-analogue-scale method when they were shown each of 8 different odors, 8 different visual images, or combinations of these different types of visual images and odors. The subjects were also asked to judge how much each specific combination of odor and visual image was harmonious on a 7-point scale. The results show some averaging effects when visual and olfactory information fell into different categories (i.e., mild, sharp, clear or dense) in the impression space. On the other hand, the impression of image–odor combination tended to be enhanced when the impression of odor or image alone fell into the same category along with the condition when they were judged as being harmonious. These results suggest that the way combination of visual and olfactory information elicits the impression may depend on the impression of individual visual and olfactory information and whether they are harmonious.

With the rapid growth of tagged images, researchers are now resorting to this high semantic textual information for image clustering, which has showed higher clustering performance compared with conventional methods using the low level visual features. However, how to bridge the gap between the semantic information and the visual information is still an open problem. In this paper, a novel topic model based framework is proposed for tagged image clustering, which consists of three steps. Firstly, the statistics between the visual features and the tag features are calculated to utilize the complementary characteristic between the two sources of information. Then the new tag feature embedded by visual information is extracted as the feature of images. Finally, typical topic model, i.e., Latent Dirichlet Allocation, is applied for image clustering. The proposed method can make full use of the tag and visual information for image clustering. Experimental results on two widely used datasets, i.e., Pascal VOC 2007 and NUS-WIDE Flickr databases, demonstrate the effectiveness of the proposed method.

Unsupervised image captioning often grapples with challenges such as image–text mismatches and modality gaps, resulting in suboptimal captions. This paper introduces a semantic-enhanced cross-modal fusion model (SCFM) to address these issues. The SCFM integrates three innovative components: a text semantic enhancement network (TSE-Net) for nuanced semantic representation; contrast learning for optimizing similarity measures between text and images; and enhanced visual selection decoding (EVSD) for precise captioning. Unlike existing methods that struggle with capturing accurate semantic relationships and flexibility across scenarios, the proposed model provides a robust solution for unbiased and diverse captioning. Through experimental evaluations on the MS COCO and Flickr30k datasets, SCFM demonstrates significant improvements over the benchmark model, enhancing the CIDEr and BLEU-4 metrics by 3.6% and 3.2%, respectively. Visualization analysis further reveals the model’s superiority in increasing variability between hidden features and its potential in cross-domain and stylized image captioning. The findings not only contribute to the advancement of image captioning techniques but also open avenues for future research. Further investigations will explore SCFM’s adaptability to other multimodal tasks and refine it for more intricate image–text relationships.

In this study, we investigated the role of visual and phonological information in lexical access of Chinese characters. Homophonic English words have been the main source of stimuli for word recognition research. However, since these stimuli also often look alike, visual and phonological information may be confounded in reported experiments. In contrast, many homophonic Chinese characters are visually distinct. In addition, visually similar characters often have very different pronunciations. These characteristics allow a more controlled investigation of the roles of visual and phonological information in activation of meaning. In the present study, two types of Chinese characters were used in a semantic categorization paradigm: integrated characters, which contain strokes that are not separable; and compound characters, which contain at least two clearly identifiable components. The results show that the recognition of a Chinese integrated character depends primarily on visual information, whereas the recognition of a Chinese compound character relies on visual, phonological, and semantic information. It is concluded that visual information plays a greater role in Chinese character recognition than has previously been documented.

Automatically describing contents of an image using natural language has drawn much attention because it not only integrates computer vision and natural language processing but also has practical applications. Using an end-to-end approach, we propose a bidirectional semantic attention-based guiding of long short-term memory (Bag-LSTM) model for image captioning. The proposed model consciously refines image features from previously generated text. By fine-tuning the parameters of convolution neural networks, Bag-LSTM obtains more text-related image features via feedback propagation than other models. As opposed to existing guidance-LSTM methods which directly add image features into each unit of an LSTM block, our fine-tuned model dynamically leverages more text-conditional image features, acquired by the semantic attention mechanism, as guidance information. Moreover, we exploit bidirectional gLSTM as the caption generator, which is capable of learning long term relations between visual features and semantic information by making use of both historical and future contextual information. In addition, variations of the Bag-LSTM model are proposed in an effort to sufficiently describe high-level visual-language interactions. Experiments on the Flickr8k and MSCOCO benchmark datasets demonstrate the effectiveness of the model, as compared with the baseline algorithms, such as it is 51.2% higher than BRNN on CIDEr metric.

Automatically describing image content with natural language is a fundamental challenge for computer vision community. General methods used visual information to generate sentences directly. However, only depending on the visual information is not enough to generate the fine-grained descriptions for given images. In this paper, we exploit the fusion of visual information and high-level semantic information for image captioning. We propose a hierarchical deep neural network, which consists of the bottom layer and the top layer. The former extracts the visual and high-level semantic information from image and detected regions, respectively, while the latter integrates both of them with adaptive attention mechanism for the caption generation. The experimental results achieve the competing performances against the state-of-the-art methods on MSCOCO dataset.

Zero-shot learning (ZSL) recognizes unseen images by sharing semantic knowledge transferred from seen images, encouraging the investigation of associations between semantic and visual information. Prior works have been devoted to the alignment of global visual features with semantic information, i.e., attribute vectors, or further mining the local part regions related to each attribute and then simply concatenating them for category decisions. Although effective, these works ignore intrinsic interactions between local parts and the whole object, which enables a more discriminative and representative knowledge transfer for ZSL. In this paper, we propose a Part-Object Progressive Refinement Network (POPRNet), where discriminative and transferable semantics are progressively refined by the cooperation between parts and the whole object. Specifically, POPRNet incorporates discriminative part semantics and object-centric semantics guided by semantic intensity to improve cross-domain transferability. To achieve part-object learning, a semantic-augment transformer (SaT) is proposed to model the part-object relation at the part-level via an encoder and at the object-level via a decoder, generating a comprehensive semantic representation to boost discriminability and transferability. By introducing the prototype updating module embedded with the prototype selection layers, the discriminative ability of the updated category prototype is enhanced to further improve the recognition performance of ZSL. Extensive experiments are conducted to demonstrate the superiority and competitiveness of our proposed POPRNet method on three public benchmark datasets. The code is available at https://github.com/ManLiuCoder/POPRNet.

Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) has been proposed recently, putting the traditional Sketch-based Image Retrieval (SBIR) under the setting of zero-shot learning. Dealing with both the challenges in SBIR and zero-shot learning makes it become a more difficult task. Previous works mainly focus on utilizing one kind of information, i.e., the visual information or the semantic information. In this paper, we propose a SketchGCN model utilizing the graph convolution network, which simultaneously considers both the visual information and the semantic information. Thus, our model can effectively narrow the domain gap and transfer the knowledge. Furthermore, we generate the semantic information from the visual information using a Conditional Variational Autoencoder rather than only map them back from the visual space to the semantic space, which enhances the generalization ability of our model. Besides, feature loss, classification loss, and semantic loss are introduced to optimize our proposed SketchGCN model. Our model gets a good performance on the challenging Sketchy and TU-Berlin datasets.

Most recent arts in image captioning rely solely on exploring the information contains in the image or modeling the inner-relations among visual features, which fails to generate informative captions in some cases. Part of what defines humans is the ability of common-sense reasoning behind semantic association, which is different from machines. To this end, we propose a Common-Sense Aware method (CSA) for image captioning, which capitalizes general prior knowledge to associate extra semantic information during generation to infer more informative captions. Specifically, based on ConceptNet, we extract common-sense knowledge features using pre-generated concepts to provide comprehensive associated semantic information for captioning. We conduct extensive experiments on the MS COCO dataset to demonstrate the effectiveness of CSA, results show that it furthers state-of-the-arts.KeywordsCommon-sense perceptionConceptNetAttention mechanismImage captioning

The field of Artificial Intelligence had made a huge change in all sectors of computer technology. It had given a comfort zone and user friendly environment for human beings. Similarly the most trending topic also made a lots of improvement in Image captioning. Image captioning is most widely used technology in computer field. In this paper we will be implementing generation of image captions using neural network models and understand what the role of this models is in caption generation. The models used for implementation are CNN, RNN with LSTM. In this paper the dataset used is Flickr 8k dataset which consist images of different varieties and different activities performed by human beings. The main aim is to extract all the features from the given input image and establish an appropriate relationship from the extracted features and tried to get the most accurate captions for given input images.

A multi-modal 3D object-detection method, based on data from cameras and LiDAR, has become a subject of research interest. PointPainting proposes a method for improving point-cloud-based 3D object detectors using semantic information from RGB images. However, this method still needs to improve on the following two complications: first, there are faulty parts in the image semantic segmentation results, leading to false detections. Second, the commonly used anchor assigner only considers the intersection over union (IoU) between the anchors and ground truth boxes, meaning that some anchors contain few target LiDAR points assigned as positive anchors. In this paper, three improvements are suggested to address these complications. Specifically, a novel weighting strategy is proposed for each anchor in the classification loss. This enables the detector to pay more attention to anchors containing inaccurate semantic information. Then, SegIoU, which incorporates semantic information, instead of IoU, is proposed for the anchor assignment. SegIoU measures the similarity of the semantic information between each anchor and ground truth box, avoiding the defective anchor assignments mentioned above. In addition, a dual-attention module is introduced to enhance the voxelized point cloud. The experiments demonstrate that the proposed modules obtained significant improvements in various methods, consisting of single-stage PointPillars, two-stage SECOND-IoU, anchor-base SECOND, and an anchor-free CenterPoint on the KITTI dataset.