D-CNN: A New model for Generating Image Captions with Text Extraction Using Deep Learning for Visually Challenged Individuals

Heterogeneous data have become a key issue restricting the monitoring accuracy of volcanic ash cloud and rapid application of remote sensing. In view of the characteristics of classification and heterogeneous data in volcanic ash cloud monitoring, a monitoring method of volcanic ash cloud using feature fusion, convolutional neural networks (CNN) and long short-term memory (LSTM) (FF–CNN–LSTM) is presented in this paper. In this method, firstly, the target features in image are identified by CNN and the time sequence information in text is extracted by LSTM. Secondly, the mapping relationship between text information and image features is learned by fusing image target and text high-level features, and then the identification and diffusion of volcanic ash cloud from heterogeneous data containing only image and text were performed. Finally, the presented FF–CNN–LSTM method is tested by the simulation experiment and the true heterogeneous data of Enta volcanic ash cloud case. The experimental results show that compared with the single CNN, LSTM and the simple combination between CNN and LSTM (CNN–LSTM), the presented FF–CNN–LSTM method in this paper can be fitted with less training steps and has high accuracy and low loss rate; the obtained distribution of volcanic ash cloud is clear and intuitive and has the characteristics with fewer model parameters, simple calculation and high accuracy. It also shows the feasibility and effectiveness of the presented method in volcanic ash cloud monitoring to some extent. The results reveal that the presented method can potentially contribute the monitoring of volcanic ash cloud and disaster prevention and mitigation.

In a typical image captioning pipeline, a Convolutional Neural Network (CNN) is used as the image encoder and Long Short-Term Memory (LSTM) as the language decoder. LSTM with attention mechanism has shown remarkable performance on sequential data including image captioning. LSTM can retain long-range dependency of sequential data. However, it is hard to parallelize the computations of LSTM because of its inherent sequential characteristics. In order to address this issue, recent works have shown benefits in using self-attention, which is highly parallelizable without requiring any temporal dependencies. However, existing techniques apply attention only in one direction to compute the context of the words. We propose an attention mechanism called Bi-directional Self-Attention (Bi-SAN) for image captioning. It computes attention both in forward and backward directions. It achieves high performance comparable to state-of-the-art methods.

Generating image captions is a very important task in the field of AI. The rise of deep learning methods has driven the development of advanced models that merge image feature extraction with natural language processing, enabling systems to generate captions that are both coherent and meaningful. Image captioning requires identifying key objects, understanding their relationships, and creating a description that flows naturally, with proper grammar. Our approach uses a convolutional neural network (CNN) for extracting visual features and a long short-term memory (LSTM) network for generating natural language captions. The model is trained and evaluated on the Flickr8k dataset. Additionally, beam search and T5 refinement have been included to enhance fluency and coherence.

VD-SAN: Visual-Densely Semantic Attention Network for Image Caption Generation

Online leakage current classification using convolutional neural network long short-term memory for high voltage insulators on web-based service

Contextual information contained within human environments, such as text on signs, symbols and objects provide important information for robots to use for exploration and navigation. To identify and segment contextual information from images obtained in these environments data-driven methods such as Convolutional Neural Networks (CNNs) can be used. However, these methods require significant amounts of human labeled data which is time-consuming to obtain. In this paper, we present the novel Weakly Supervised Mask Data Distillation (WeSuperMaDD) architecture for autonomously generating pseudo segmentation labels (PSLs) using CNNs not specifically trained for the task of text segmentation, e.g., CNNs alternatively trained for: object classification or image captioning. WeSuperMaDD is uniquely able to generate PSLs using learned image features from datasets that are sparse and with limited diversity, which are common in robot navigation tasks in human-centred environments (i.e., malls, stores). Our proposed architecture uses a new mask refinement system which automatically searches for the PSL with the fewest foreground pixels that satisfies cost constraints. This removes the need for handcrafted heuristic rules. Extensive experiments were conducted to validate the performance of WeSuperMaDD in generating PSLs for datasets containing text of various scales, fonts, orientations, curvatures, and perspectives in several indoor/outdoor environments. A detailed comparison study conducted with existing approaches found a significant improvement in PSL quality. Furthermore, an instance segmentation CNN trained using the WeSuperMaDD architecture achieved measurable improvements in accuracy when compared to an instance segmentation CNN trained with Naïve PSLs. We also found our method to have comparable performance to existing text detection methods.

Generating a novel and descriptive caption of an image is drawing increasing interests in computer vision, natural language processing, and multimedia communities. In this work, we propose an end-to-end trainable deep bidirectional LSTM (Bi-LSTM (Long Short-Term Memory)) model to address the problem. By combining a deep convolutional neural network (CNN) and two separate LSTM networks, our model is capable of learning long-term visual-language interactions by making use of history and future context information at high-level semantic space. We also explore deep multimodal bidirectional models, in which we increase the depth of nonlinearity transition in different ways to learn hierarchical visual-language embeddings. Data augmentation techniques such as multi-crop, multi-scale, and vertical mirror are proposed to prevent overfitting in training deep models. To understand how our models “translate” image to sentence, we visualize and qualitatively analyze the evolution of Bi-LSTM internal states over time. The effectiveness and generality of proposed models are evaluated on four benchmark datasets: Flickr8K, Flickr30K, MSCOCO, and Pascal1K datasets. We demonstrate that Bi-LSTM models achieve highly competitive performance on both caption generation and image-sentence retrieval even without integrating an additional mechanism (e.g., object detection, attention model). Our experiments also prove that multi-task learning is beneficial to increase model generality and gain performance. We also demonstrate the performance of transfer learning of the Bi-LSTM model significantly outperforms previous methods on the Pascal1K dataset.

The way towards generating a website front end involves a designer settling on an idea for what kind of layout they want the website to have, then proceeding to plan and implement each aspect one by one until they have converted what they initially laid out into its Html front end form, this process can take a considerable time, especially considering the first draft of the design is traditionally never the final one. This process can take up a large amount of resource real estate, and as we have laid out in this paper, by using a Model consisting of various Neural Networks trained on a custom dataset. It can be automated into assisting designers, allowing them to focus on the other more complicated parts of the system they are designing by quickly generating what would rather be straightforward busywork. Over the past 20 years, the boom in how much the internet is used and the sheer volume of pages on it demands a high level of work and time to create them. For the efficiency of the process, we proposed a multi-model-based architecture on image captioning, consisting of Convolutional neural network (CNN) and Long short-term memory (LSTM) models. Our proposed approach trained on our custom-made database can be automated into assisting designers, allowing them to focus on the other more complicated part of the system. We trained our model in several batches over a custom-made dataset consisting of over 6300 files and were finally able to achieve a Bilingual Evaluation Understudy (BLEU) score for a batch of 50 hand-drawn images at 87.86%

Image captioning often requires a large set of training image-sentence pairs. In practice, however, acquiring sufficient training pairs is always expensive, making the recent captioning models limited in their ability to describe objects outside of training corpora (i.e., novel objects). In this paper, we present Long Short-Term Memory with Copying Mechanism (LSTM-C) — a new architecture that incorporates copying into the Convolutional Neural Networks (CNN) plus Recurrent Neural Networks (RNN) image captioning framework, for describing novel objects in captions. Specifically, freely available object recognition datasets are leveraged to develop classifiers for novel objects. Our LSTM-C then nicely integrates the standard word-by-word sentence generation by a decoder RNN with copying mechanism which may instead select words from novel objects at proper places in the output sentence. Extensive experiments are conducted on both MSCOCO image captioning and ImageNet datasets, demonstrating the ability of our proposed LSTM-C architecture to describe novel objects. Furthermore, superior results are reported when compared to state-of-the-art deep models.

COVID-19 pandemic has indeed plunged the global community especially African countries into an alarming difficult situationculminating into a great deal amounts of catastrophes such as economic recession, political instability and loss of jobs. Thepandemic spreads exponentially and causes loss of lives. Following the outbreak of the omicron new variant of concern,forecasting and identification of the COVID-19 infection cases is very vital for government at various levels. Hence, havingknowledge of the spread at a particular point in time, swift actions can be taken by government at various levels with a view toaccordingly formulate new policies and modalities towards minimizing the trajectory of the consequences of COVID-19 pandemicto both public health and economic sectors. Here, a potent combination of Convolutional Neural Network (CNN) learning algorithm along with Long Short Term Memory(LSTM) learning algorithm has been proposed in this work in order to produce a hybrid of a deep learning algorithm ConvolutionalNeural Network - Long Short Term Memory (CNN-LSTM) for forecasting COVID-19 infection cases particularly in Nigeria, SouthAfrica and Botswana. Forecasting models for COVID-19 infection cases in Nigeria, South Africa and Botswana, were developedfor 10 days using deep learning-based approaches namely CNN, LSTM and CNN-LSTM deep learning algorithm respectively. The models were evaluated on the basis of four standard performance evaluation metrics which include accuracy, MSE, MAEand RMSE respectively. However, the CNN-LSTM deep learning-based forecasting model achieved the best accuracy of98.30%, 97.60%, and 97.74% for Nigeria, South Africa and Botswana respectively; and in the same manner, achieved lesserMSE, MAE and RMSE values compared to models developed with CNN and LSTM respectively. Taken together, the CNN-LSTM deep learning-based forecasting model for COVID-19 infection cases in Nigeria, South Africaand Botswana dramatically surpasses the two other DL based forecasting models (CNN and LSTM) for COVID-19 infectioncases in Nigeria, South Africa and Botswana in terms of not only the best accuracy of with 98.30%, 97.60%, and 97.74% but alsoin terms of lesser MSE, MAE and RMSE.

This research aims to implement deep learning techniques to determine fact and fake news in Indonesian language. The methods used are Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM). The stages of the research consisted of collecting data, labeling data, preprocessing data, word embedding, splitting data, forming CNN and LSTM models, evaluating, testing new input data and comparing evaluations of the established CNN and LSTM models. The Data are collected from a fact and fake news provider site that is valid, namely TurnbackHoax.id. There are 1786 news used in this study, with 802 fact and 984 fake news. The results indicate that the CNN and LSTM methods were successfully applied to determine fact and fake news in Indonesian language properly. CNN has an accuracy test, precision and recall value of 0.88, while the LSTM model has an accuracy test and precision value of 0.84 and a recall of 0.83. In testing the new data input, all of the predictions obtained by CNN are correct, while the prediction results obtained by LSTM have 1 wrong prediction. Based on the evaluation results and the results of testing the new data input, the model produced by the CNN method is better than the model produced by the LSTM method.

Although image captioning technology has made great progress in recent years, the quality of Chinese image description is far from enough. In this paper, we focus on the problem of Chinese image captioning with the aim to improve the quality of Chinese image description. A novel framework for Chinese image captioning based on image label information (CIC) is proposed in this paper. Firstly, image label information is extracted by a multi-layer model with shortcut connections. Then the label information is input into the neural network with an extension of LSTM, which we coin L-LSTM for short, to generate the Chinese image descriptions. Extensive experiments are conducted on various image caption datasets such as Flickr8k-cn, Flickr30 k-cn. The experimental results verify the effectiveness of the proposed framework (CIC). It obtains 27.1% and 21.2% BLEU4 average values of Flickr8k-cn and Flickr30k-cn, respectively, which outperforms the state-of-art model in Chinese image captioning domain.

Visual attention has been successfully applied in image captioning to selectively incorporate the most relevant areas to the language generation procedure. However, the attention in current image captioning methods is only guided by the hidden state of language model, e.g. LSTM (Long-Short Term Memory), indirectly and implicitly, and thus the attended areas are weakly relevant at different time steps. Besides the spatial relationship of attention areas, the temporal relationship in attention is crucial for image captioning according to the attention transmission mechanism of human vision. In this paper, we propose a new spatio-temporal memory attention (STMA) model to learn the spatio-temporal relationship in attention for image captioning. The STMA introduces the memory mechanism to the attention model through a tailored LSTM, where the new cell is used to memorize and propagate the attention information, and the output gate is used to generate attention weights. The attention in STMA transmits with memory adaptively and dependently, which builds strong temporal connections of attentions and learns the spatio-temporal relationship of attended areas simultaneously. Besides, the proposed STMA is flexible to combine with attention-based image captioning frameworks. Experiments on MS COCO dataset demonstrate the superiority of the proposed STMA model in exploring the spatio-temporal relationship in attention and improving the current attention-based image captioning.

Long short-term memory (LSTM) networks are a tweaked version of the recurrent neural networks (RNN) that enable information persistence and fall under the deep learning domain. It’s widely used in various applications, such as air pollution forecasting, flood forecasting, handwriting generation, language modeling, image captioning, question answering, video to text conversion, machine translation, etc. LSTM networks are used to process sequential data that involves the temporal correlation between a given data segment and its previous segment. The chapter starts with a discussion of the LSTM architecture, its variants, and its applications across various domains. The chapter also provides a comprehensive discussion of various types of LSTMs being used to solve the problem of air pollution forecasting, along with discussing a general pipeline used for the same.

The automatic narration of a natural scene is an important trait in artificial intelligence that unites computer vision and natural language processing. Caption generation is a challenging task in scene understanding. Most of the state-of-the-art methods are using deep convolutional neural network models to extract visual features of the entire image, based on which the parallel structures between images and sentences are exploited using recurrent neural networks for image captioning. However, in such models, only visual features are exploited for caption generation. This work investigated that fusion of text available in an image can give more fined-grained captioning of a scene. In this paper, we have proposed a model which incorporates a deep convolutional neural network and long short-term memory to boost the accuracy of image captioning by fusing text feature available in an image with the visual features extracted in state-of-the-art methods. We have validated the effectiveness of the proposed model on the benchmark datasets (Flickr8k and Flickr30k). The experimental outcomes illustrate that the proposed model outperformed the state-of-the-art methods for image captioning.