Image captioning in Hindi language using transformer networks

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Image captioning refers to the process of generating a textual description for an image which defines the object and activity within the image. It is an intersection of computer vision and natural language processing where computer vision is used to understand the content of an image and language modelling from natural language processing is used to convert an image into words in the right order. A large number of works exist for generating image captioning in English language, but no work exists for generating image captioning in Hindi language. Hindiis the official language of India, and it is the fourth most-spoken language in the world, after Mandarin, Spanish and English. The current paper attempts to bridge this gap. Here an attention-based novel architecture for generating image captioning in Hindi language is proposed. Convolution neural network isused as an encoder to extract features from an input image and gated recurrent unit based neural network isused as a decoder to perform language modelling up to the word level. In between, we have used the attention mechanism which helps the decoder to look into the important portions of the image. In order to show theefficacy of the proposed model, we have first created a manually annotated image captioning training corpusin Hindi corresponding to popular MS COCO English dataset having around 80000 images. Experimental results show that our proposed model attains a BLEU1score of 0.5706 on this data set.

In this paper, alternative direction finding methods have been proposed using deep learning techniques. Firstly, Regeression and Classification models have created by using Convolutional Neural Networks (CNNs). In the second Convolutional Neural Networks and Recurrent Neural Networks (RNNs) have been utilized in the proposed methods. Despite having vast amount of direction finding studies, utilization of neural networks is scarce in literature and past works mostly only includes usage of CNNs. In this study, direction finding is performed by learning signals reaching multiple antenna arrays by networks. Created neural networks have been fed with different data formats and their performances against noisy and no-noise data have been shown. In addition, comparative analysis of the developed models were made in the similar Signal-to-Noise Ratio (SNR) range with the subspace based MUSIC algorithm, which is frequently used in direction finding.

The natural language processing (NLP) area has been magically transformed by the Deep Neural Network (DNN). The two variations of Neural Networks, Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) can handle the different NLP tasks effectively. CNN is based on feature extraction using N-grams at higher levels. The RNN can be effectively used to model the sequential information. Choosing a neural technique for the various NLP tasks is always a challenge. The paper focuses on the evaluating the performance of the two DNN techniques CNN and RNN for the various NLP tasks, so that appropriate technique can be selected.

Modeling accurate text recognizers for scanned Devanagari scripts is a challenging task, primarily due to the ingrained complexities of characters, vowels, conjuncts, and modifiers in the word formation of the script. This paper presents an attention-based convolutional recurrent neural network (CRNN) for improving the text recognition accuracy of the Devanagari script images. The proposed model is an encoder–decoder network with an attention mechanism. The encoder is a convolutional neural network (CNN), which extracts the significant and high-level features from the input images. The encoder also generates the extracted features in an ordered sequence. The decoder uses a bidirectional long short-term (BLSTM) network with an attention mechanism, followed by a connectionist temporal classification (CTC) layer to predict the text in images. The attention mechanism enables the decoder to selectively exploit the high-level features of the encoder congruently. We have trained, validated, and tested the performance of the proposed model on three datasets, viz. publicly available IIIT-H scene image dataset, synthetic dataset, and real dataset curated during this work. We have also compared the proposed model results with work already attempted in the area for the three datasets. We found that the performance of our proposed model outdoes the outcomes obtained from other models on the three datasets.

Convolutional neural network (CNN) and recurrent neural network (RNN) are two popular architectures used in text classification. Traditional methods to combine the strengths of the two networks rely on streamlining them or concatenating features extracted from them. In this paper, we propose a novel method to keep the strengths of the two networks to a great extent. In the proposed model, a convolutional neural network is applied to learn a 2D weight matrix where each row reflects the importance of each word from different aspects. Meanwhile, we use a bidirectional RNN to process each word and employ a neural tensor layer that fuses forward and backward hidden states to get word representations. In the end, the weight matrix and word representations are combined to obtain the representation in a 2D matrix form for the text. We carry out experiments on a number of datasets for text classification. The experimental results confirm the effectiveness of the proposed method.

Neural network models have recently been applied to the task of automatic essay scoring, giving promising results. Existing work used recurrent neural networks and convolutional neural networks to model input essays, giving grades based on a single vector representation of the essay. On the other hand, the relative advantages of RNNs and CNNs have not been compared. In addition, different parts of the essay can contribute differently for scoring, which is not captured by existing models. We address these issues by building a hierarchical sentence-document model to represent essays, using the attention mechanism to automatically decide the relative weights of words and sentences. Results show that our model outperforms the previous state-of-the-art methods, demonstrating the effectiveness of the attention mechanism.

A hybrid model based on neural networks for biomedical relation extraction

Caption generation requires best of both Computer Vision and Natural Language Processing. Due to recent improvements in both of them many efficient models have been developed. Automatic Image Captioning can be utilized to provide descriptions of website content or to engender frame-by-frame descriptions of video for the vision-impaired and in many such applications. In this work, a model is described which is utilized to generate novel image captions for a previously unseen image by utilizing a multimodal architecture by amalgamation of a Recurrent Neural Network (RNN) and a Convolutional Neural Network (CNN). The model is trained on Microsoft Common Objects in Context (MSCOCO), an image captioning dataset that aligns captions and images in the same representation space, so that an image is close to its relevant captions in that space and far away from dissimilar captions and dissimilar images. ResNet-50 architecture is used for extracting features from the images and GloVe embeddings are used along with Gated Recurrent Unit (GRU) in Recurrent Neural Network (RNN) for text representation. MSCOCO evaluation server is used for evaluation of the machine generated caption for a given image.

To develop a natural language processing system that identifies relations of medications with adverse drug events from clinical narratives. This project is part of the 2018 n2c2 challenge. We developed a novel clinical named entity recognition method based on an recurrent convolutional neural network and compared it to a recurrent neural network implemented using the long-short term memory architecture, explored methods to integrate medical knowledge as embedding layers in neural networks, and investigated 3 machine learning models, including support vector machines, random forests and gradient boosting for relation classification. The performance of our system was evaluated using annotated data and scripts provided by the 2018 n2c2 organizers. Our system was among the top ranked. Our best model submitted during this challenge (based on recurrent neural networks and support vector machines) achieved lenient F1 scores of 0.9287 for concept extraction (ranked third), 0.9459 for relation classification (ranked fourth), and 0.8778 for the end-to-end relation extraction (ranked second). We developed a novel named entity recognition model based on a recurrent convolutional neural network and further investigated gradient boosting for relation classification. The new methods improved the lenient F1 scores of the 3 subtasks to 0.9292, 0.9633, and 0.8880, respectively, which are comparable to the best performance reported in this challenge. This study demonstrated the feasibility of using machine learning methods to extract the relations of medications with adverse drug events from clinical narratives.

This paper proposes a novel deep learning architecture involving combinations of Convolutional Neural Networks (CNN) layers and Recurrent neural networks (RNN) layers that can be used to perform segmentation and classification of 5 cardiac rhythms based on ECG recordings. The algorithm is developed in a sequence to sequence setting where the input is a sequence of five second ECG signal sliding windows and the output is a sequence of cardiac rhythm labels. The novel architecture processes as input both the spectrograms of the ECG signal as well as the heartbeats' signal waveform. Additionally, we are able to train the model in the presence of label noise. The model's performance and generalizability is verified on an external database different from the one we used to train. Experimental result shows this approach can achieve an average F1 scores of 0.89 (averaged across 5 classes). The proposed model also achieves comparable classification performance to existing state-of-the-art approach with considerably less number of training parameters.

Deep Learning and Beyond: AI’s New Horizons” is an expansive monograph that delves into the dynamic and rapidly evolving field of deep learning, a key area of artificial intelligence. This comprehensive work provides an insightful exploration of the revolutionary developments, foundational concepts, and future directions of deep learning, positioning itself as a crucial resource for understanding this transformative technology. The monograph commences with an introductory chapter that lays the groundwork by elucidating the background of machine learning and chronicling the ascent of deep learning. This foundational perspective is vital for comprehending the underlying principles and methodologies that have propelled deep learning to the forefront of technological innovation. Subsequent chapters delve into the “Historical Evolution” of AI, tracing the origins of neural networks from their early inception to the Renaissance period of deep learning. This historical analysis is pivotal in understanding the field’s trajectory and significant milestones. The “Foundations of Deep Learning” chapter presents an in-depth analysis of neural networks, covering their basic architecture, activation functions, layers, and the critical process of backpropagation and gradient descent. This is followed by examining “Advanced Neural Network Architectures,” where the focus shifts to state-of-the-art models such as convolutional neural networks, recurrent neural networks, and groundbreaking transformers and attention mechanisms. In addressing the “Training Deep Models,” the monograph explores regularization techniques, optimization strategies, and the increasingly significant role of transfer learning and pre-trained models. The application-oriented chapter, “Applications of Deep Learning,” showcases real-world implementations in domains like image recognition, natural language viii processing, and autonomous systems, illustrating deep learning technologies’ vast potential and versatility. The monograph also tackles the “Challenges in Deep Learning,” including overfitting, interpretability, and the ethical considerations surrounding AI deployment. It then ventures into the “Future of Deep Learning,” contemplating emerging trends such as integrating quantum computing, neural architecture search, and the convergence of neuroscience and deep learning. “Case Studies” provide practical insights and empirical evidence of the impact of deep learning. At the same time, the concluding chapter synthesizes the current state of deep learning, offers predictions for the future, and provides reflective thoughts on this transformative field. “Deep Learning and Beyond: AI’s New Horizons” educates and informs readers about the vast potential and future possibilities of deep learning and artificial intelligence. Keywords: Deep Learning, Artificial Intelligence, Neural Networks, Machine Learning, Convolutional Neural Networks, Recurrent Neural Networks, Transformers, Quantum Computing, Ethical AI, Neural Architecture Search, Natural Language Processing, Autonomous Systems, AI in Healthcare, Future of AI, AI Applications, AI Challenges.

In this digital era, there is an exponential growth of text-based content in the electronic world. Data as texts exist in the form of documents, social media posts on Facebook, Twitter, etc., logs, sensor data, and emails. Twitter is a social platform where users express their views on various aspects in a day to day life. Twitter produces over 500 million tweets daily that is 6000 tweets per second. Twitter data is, by definition, very noisy and unstructured in nature. Text classifications based on the machine learning techniques have problems like poor generalization ability and sparsity dimension explosion. Classifiers based on deep learning techniques are implemented to improve accuracy to overcome shortcomings of machine learning techniques and to avoid feature extraction processes and have high prediction accuracy and strong learning ability. In this work, the classification of tweets is performed on Covid-19 dataset by implementing deep learning techniques namely Convolution Neural Network (CNN), Recurrent Neural Network (RNN), Recurrent Convolution Neural Network (RCNN), Recurrent Neural Network with Long Short Term Memory (RNN+LSTM), and Bidirectional Long Short Term Memory with Attention (BI-LSTM + Attention). The algorithms are implemented using two-word embedding techniques namely Global Vectors for Word Representation (GloVe) and Word2Vec. RNN with Bidirectional LSTM model has performed better than all the classifiers considered. It has classified the text with an accuracy of 93% and above when used with GloVe and Word2Vec.

The highly structured nature of medical reports makes them feasible for automated large-scale patient identification. This study aimed to develop a natural language processing (NLP) model to retrospectively retrieve patients with presence and history of carotid stenosis (CS) using their ultrasound reports. Ultrasound reports from our institution between January 2016 and December 2017 were selected. To process the texts, we developed a parser to divide the raw text into fields. For baseline method, we used bag-of-n-grams and term frequency inverse document frequency as the features and used linear classifiers. Logistic regression was performed as the baseline model. Convolution and recurrent neural networks (CNN; RNN) with attention mechanism were applied to the dataset to improve the classification accuracy. We had 1220 ultrasound reports for training and 307 for testing, totaling to 1527 reports. For predicting history of CS, both CNN and RNN-attention models had a significantly higher specificity than logistic regression. In addition, RNN-attention also had a significantly higher F1 score and accuracy. For predicting presence of carotid stenosis, all models achieved above 93% accuracy. RNN-attention achieved a 95.4% accuracy, although the difference with logistic regression was not statistically significant. RNN-attention had a statistically significant higher specificity than logistic regression. We developed linear, CNN, and RNN models to predict history and presence of CS from ultrasound reports. We have demonstrated NLP to be an efficient, accurate approach for large-scale retrospective patient identification, with applications in long-term follow-up of patients and clinical research studies. • Natural language processing models using both linear classifiers and neural networks can achieve a good performance, with an overall accuracy above 90% in predicting history and presence of carotid stenosis. • Convolution and recurrent neural networks, especially with additional features including field awareness and attention mechanism, have superior performance than traditional linear classifiers. • NLP is shown to be an efficient approach for large-scale retrospective patient identification, with applications in long-term follow-up of patients and further clinical research studies.

Deep learning such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) have been successfully applied to Natural Language Processing (NLP). Previous research shows that RNN and CNN achieved significant results in extracting long and short-term dependencies. However, there is a lack of research on Persian/Dari due to low resources and with many difficulties in preprocessing. In this research paper, we proposed a novel sentiment analysis model using bidirectional GRU and CNN to address the challenges and improve sentiment knowledge in the Persian/Dari language. Firstly, a preprocessor is utilized to enhance data quality by eliminating noise. Secondly, a bidirectional Gated Recurrent Unit is used to capture long-term dependencies. Thirdly, our model(named PersiSentNet) employs CNN with max pooling to extract contextual features and reduce feature dimensionality. Gaussian Noise and Dropout are applied as regularization techniques to avoid the overfitting problem. Finally, the model’s performance is verified on two standard datasets, and the experimental results indicate that our model significantly outperforms the state-of-the-art models, highlighting its robustness and effectiveness in addressing the challenges of Persian/Dari language sentiment analysis.